oldwiki.scinet.utoronto.ca - User contributions [en-gb]

Python

2014-09-16T20:07:58Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1
| -
| -
| -
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1
| 2.1.3
| 2.2.0
| 2.3.0
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
|
|
|
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1
| 2.6.2
| 2.7.1
| -
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15.1
| 0.18
| 0.19.1
| 0.20.1
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2
| 1.2.1
| 1.3.0
| 1.3.0
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| 0.6c11
| 1.1
| 5.1
| Enables easy installation of new python modules
|-
| [http://pandas.pydata.org/ pandas]
| 0.13.0
| 0.13.0
| 0.13.0
| -
| high-performance, easy-to-use data structures and data analysis tools.
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T20:05:08Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1
| -
| -
| -
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1
| 2.1.3
| 2.2.0
| 2.3.0
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
|
|
|
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1
| 2.6.2
| 2.7.1
| -
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15.1
| 0.18
| 0.19.1
| 0.20.1
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2
| 1.2.1
| 1.3.0
| 1.3.0
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| 0.6c11
| 1.1
| 5.1
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T19:59:15Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1
| -
| -
| -
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1
| 2.1.3
| 2.2.0
| 2.3.0
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1 (Python 2.7.2)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15 (Python 2.7.x)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2 (Python 2.7.2)
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T19:57:38Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1
| 2.4.0
| 3.0.0
| 3.1.1
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8
| 1.0.4
| 1.1.1
| 1.1.0
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1 (Python 2.7.2)
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1 (Python 2.7.2)
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1 (Python 2.7.2)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15 (Python 2.7.x)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2 (Python 2.7.2)
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T19:54:36Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0
| 1.2.0
| 1.3.0
| 1.3.1
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1 (Python 2.7.2)
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8 (Python 2.7.2)
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1 (Python 2.7.2)
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1 (Python 2.7.2)
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1 (Python 2.7.2)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15 (Python 2.7.x)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2 (Python 2.7.2)
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T19:52:28Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11
| 0.13.1
| 1.0.0
| 1.2.1
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0 (Python 2.7.2)
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1 (Python 2.7.2)
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8 (Python 2.7.2)
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1 (Python 2.7.2)
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1 (Python 2.7.2)
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1 (Python 2.7.2)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15 (Python 2.7.x)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2 (Python 2.7.2)
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T19:51:02Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| -
| -
| -
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2
| 2.5.3
| 2.5.5
| -
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11 (Python 2.7.2)
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0 (Python 2.7.2)
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1 (Python 2.7.2)
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8 (Python 2.7.2)
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1 (Python 2.7.2)
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1 (Python 2.7.2)
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1 (Python 2.7.2)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15 (Python 2.7.x)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2 (Python 2.7.2)
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T19:49:26Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
|
|
|
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2 (Python 2.7.2)
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11 (Python 2.7.2)
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0 (Python 2.7.2)
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1 (Python 2.7.2)
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8 (Python 2.7.2)
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1 (Python 2.7.2)
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1 (Python 2.7.2)
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1 (Python 2.7.2)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15 (Python 2.7.x)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2 (Python 2.7.2)
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T19:48:25Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2
| 1.2.2
| 1.2.2
| 1.2.2
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0
| 2.0.1
| 2.2.1
| 2.4_rc2
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2 (Python 2.7.2)
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11 (Python 2.7.2)
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0 (Python 2.7.2)
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1 (Python 2.7.2)
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8 (Python 2.7.2)
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1 (Python 2.7.2)
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1 (Python 2.7.2)
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1 (Python 2.7.2)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15 (Python 2.7.x)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2 (Python 2.7.2)
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Python

2014-09-16T19:38:22Z

Brelier: /* Modules installed system-wide */

[http://www.python.org/ Python] is programing language that continues to grow in popularity for scientific computing. It is very fast to write code in, but the software that results is much much slower than C or Fortran; one should be wary of doing too much compute-intensive work in Python.

There is a dizzying amount of documentation available for programming in Python on the [http://python.org/ Python.org webpage]; SciNet has given a mini-course of 8 lectures on [[Research Computing with Python]] in the Fall of 2013.
An excellent set of material for teaching scientists to program in Python is also available at the [http://software-carpentry.org/4_0/python/ Software Carpentry homepage].

__FORCETOC__

== Python on the GPC ==

We currently have python 2.7.2 installed, compiled against fast intel math libraries. To use this version,
<source lang=bash>
module load gcc intel python
</source>

== Modules installed system-wide ==

Many optional packages are available for Python which greatly extend the language adding important new functionality. Those packages which are likely to be important to all of our users — eg, [http://numpy.scipy.org/ NumPy], [http://www.scipy.org/ SciPy], and [http://matplotlib.sourceforge.net/ Matplotlib] are installed system-wide.

Below is a list of the packages currently installed system-wide.

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Module
!{{Hl2}}| python/2.7.2
!{{Hl2}}| python/2.7.3
!{{Hl2}}| python/2.7.5
!{{Hl2}}| python/3.3.4
!{{Hl2}}| Comments
|-
|[http://www.scipy.org/ SciPy]
| 0.10.0
| 0.11.0
| 0.14.0
| 0.14.0
| An Open-source software for mathematics, science, and engineering. Version in Python 2.7.x is linked against very fast MKL numerical libraries.
|-
|[http://numpy.scipy.org/ NumPy]
| 1.6.1
| 1.7.0
| 1.7.0
| 1.8.1
| NumPy is the fundamental package needed for scientific computing with Python. Contains fast arrays, tools for integrating C/C++ and Fortran code, linear algebra solvers, etc. SciPy is built on top of NumPy.
|-
| [http://mpi4py.scipy.org/ mpi4py]
| 1.2.2 (Python 2.7.2)
| A pythonic interface to mpi. Available with openmpi; must load an openmpi module for this to work. (There is an issue with openmpi 1.4.x + infiniband, however it does appear to work fine with IntelMPI)
|-
| [http://www.scipy.org/SciPyPackages/NumExpr Numexpr]
| 2.0 (Python 2.7.2)
| Fast, memory-efficient elementwise operations on Numpy arrays.
|-
| [http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ ScientificPython]
| 2.8
| A collection of scientific python utilities. Does not include MPI support.
|-
| [http://yt.enzotools.org/ yt]
| 2.2 (Python 2.7.2)
| A collection of python tools for analyzing astrophysical simulation output.
|-
| [http://ipython.scipy.org/moin/ iPython]
| 0.11 (Python 2.7.2)
| An enhanced interactive python.
|-
| [http://matplotlib.sourceforge.net/ Matplotlib], pylab
| 1.1.0 (Python 2.7.2)
| Matlab-like plotting for python.
|-
| [http://www.pytables.org/moin PyTables]
| 2.3.1 (Python 2.7.2)
| Fast and efficient access to HDF5 files (and HDF5-format NetCDF4 files.) Requires the <tt>hdf5/184-p1-v18-serial-gcc</tt> module to be loaded. (<tt>hdf5/187-v18-serial-gcc</tt> on CentOS6.)
|-
| [http://code.google.com/p/netcdf4-python/ NetCDF4-python]
| 0.9.8 (Python 2.7.2)
| Python interface to NetCDF4 files. Requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module to be loaded. (<tt>netcdf/4.1.3_hdf5_serial-gcc</tt> on CentOS 6)
|-
| [http://www.pyngl.ucar.edu/Nio.shtml pyNIO]
| 1.4.1 (Python 2.7.2)
| Yet another Python interface to NetCDF4 files; again, requires the <tt>netcdf/4.0.1_hdf5_v18-serial.shared-nofortran</tt> module.
|-
| [http://alfven.org/wp/hdf5-for-python/ h5py]
| 2.0.1 (Python 2.7.2)
| Yet another Python interface to HDF5 files; again, requires an HDF5 module to be loaded.
|-
| [http://pysvn.tigris.org/ PySVN]
| 1.7.1
| Python interface to the svn version control system. Requires the <tt>svn</tt> module to be loaded on CentOS5.
|-
| [http://mercurial.selenic.com/ Mercurial]
| 2.0.1 (Python 2.7.2)
| A distributed version-control system written in Python.
|-
| [http://cython.org/ Cython]
| 0.15 (Python 2.7.x)
| Cython is a compiler which compiles Python-like code files to C code and allows them to be easily called from Python.
|-
| [http://code.google.com/p/python-nose/ nose]
| 1.1.2 (Python 2.7.2)
| A unit-testing framework for python.
|-
| [http://pypi.python.org/pypi/setuptools setuptools]
| 0.6c11
| Enables easy installation of new python modules
|-
|}

== Installing your own Python Modules ==

Python provides an easy way for users to install the libraries they need in their home directories rather than having them installed system-wide. There are so many optional packages for Python people could potentially want (see e.g. http://pypi.python.org/pypi), that we recommend users install these additional packages locally in their home directories. This is almost certainly the easiest way to deal with the wide range of packages, ensure they're up to date, and ensure that users' package choices don't conflict.

To install your own Python modules, follow the instructions below. Where the instructions say <tt>python2.X</tt>, type <tt>python2.6</tt> or <tt>python2.7</tt> depending on the version of python you are using.

* First, create a directory in your home directory, <tt>${HOME}/lib/python2.X/site-packages</tt>, where the packages will go.
* Next, in your <tt>.bashrc</tt>, *after* you <tt>module load python</tt> and in the "GPC" section, add the following line:
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages/
</source>

* Re-load the modified .bashrc by typing <tt>source ~/.bashrc</tt>.

* Now, if it's a standard python package and instructions say that you can use easy_intall to install it,
** install with the following command. where <tt>packagename</tt> is the name of the package you are installing:
<source lang=bash>
easy_install --prefix=${HOME} -O1 [packagename]
</source>

** Continue doing this until all of the packages you need to install are successfully installed.
** If, upon importing the new python package, you get error messages like <tt>undefined symbol: __stack_chk_guard</tt>, you may need to use the following command instead:
<source lang=bash>
LDFLAGS=-fstack-protector easy_install --prefix=${HOME} -O1 [packagename]
</source>

* If easy_install isn't an option for your package, and the installation instructions instead talk about downloading a file and using <tt>python setup.py install</tt> then instead:
** Download the relevant files
** You will probably have to uncompress and untar them: <tt>tar -xzvf packagename.tgz</tt> or <tt>tar -xjvf packagename.bz2</tt>.
** cd into the newly created directory, and run
<source lang=bash>
python setup.py install --prefix=${HOME}
</source>

* Now, the install process may have added some .egg files or directories to your path. For each .egg directory, add that to your python path as well in your .bashrc, in the same place as you had updated PYTHONPATH before: eg,
<source lang=bash>
export PYTHONPATH=${PYTHONPATH}:${HOME}/lib/python2.X/site-packages:${HOME}/lib/python2.X/site-packages/packagename1-x.y.z-yy2.X.egg:${HOME}/lib/python2.X/site-packages/packagename2-a.b.c-py2.X.egg
</source>

* You should now be done! Now, re-source your .bashrc and test your new python modules.

* In order to keep your .bashrc relatively uncluttered, and to avoid potential conflicts among software modules, we recommend that users create their own modules (for the "module" system, not specifically python modules).

[[Brian|Here]] is an example module for the [[Brian]] package, including instructions for the installation of the python [[Brian]] package itself.

Software and Libraries

2014-09-16T18:31:17Z

Brelier: /* GPC Software */

= Software Module System =
All the software listed on this page is accessed using a modules system. This means that much of the software is not
accessible by default but has to be loaded using the module command. The
reason is that
* it allows us to easily keep multiple versions of software for different users on the system;
* it allows users to easily switch between versions.
The module system works similarly on the GPC and the TCS, although different modules are installed on these two systems.

Note that, generally, if you compile a program with a module loaded, you will have to run it with that same module loaded, to make dynamically linked libraries accessible.

{|
!{{Hl2}}|Function
!{{Hl2}}|Command
!{{Hl2}}|Comments
|-
|List available software packages:
|<pre>$ module avail</pre>
|
*If a module is not listed here, it is not supported.
*The flag "(default)" is never part of the name.
|-
|Use particular software:
|<pre> $ module load [module-name] </pre>
|
*If possible, specify only the short name (the part before the "/").
*When ambiguous, this loads the default one.
|-
|List available versions of a specific software package:
|<pre>$ module avail [short-module-name]</pre>
|
|-
|List currently loaded modules:
|<pre>$ module list</pre>
|For reproducability, it is a good idea to put this in your job scripts, so you know exactly what modules(+version) were used.
|-
|Get description of a particular module:
|<pre>$ module help [module-name]</pre>
|
|-
|Remove a module from your shell:
|<pre>$ module unload [module-name]</pre>
|
|-
|Remove all modules:
|<pre>$ module purge</pre>
|
|-
|Replace one loaded module with another:
|<pre>$ module switch [old-module-name] [new-module-name]</pre>
|
|}
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
SCINET_[short-module-name]_BASE
SCINET_[short-module-name]_LIB
SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[short-module-name]_INC}</tt> and <tt>-L${SCINET_[short-module-name]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Errors in loaded modules can arise for a few reasons, for instance:
* A module by that name may not exist.
* Some modules require other modules to have been loaded; it this requirement is not met when you try to load that module, an error message will be printed explaining what module is needed.
* Some modules cannot be loaded together: an error message will be printed explaining which modules conflict.

It is no longer recommended to load modules in the file [[Important_.bashrc_guidelines|.bashrc]] in your home directory, rather, load them explicitly on the command-line and in your job scripts.

== Default and non-default modules ==

When you load a module with its 'short' name, you will get the ''default'' version, which is the most recent (usually), recommended version of that library or piece of software. In general, using the short module name is the way to go. However, you may have code that depends on the intricacies of a non-default version. For that reason, the most common older versions are also available as modules. You can find all available modules using the <tt>module avail</tt> command.

== Naming convention ==

For modules that access applications, the full name of a module is as follows.
<pre>
[short-module-name]/[version-number]
</pre>

To have all modules conform to this convention, a number of modules' name change on Nov 3, 2010:
{|border="1" cellspacing="0" cellpadding="2"
| '''old name'''
| '''new name'''
| '''remarks'''
|-
|autoconf/autoconf-2.64     
|autoconf/2.64
|''short name unchanged''
|-
|cuda/cuda-3.0
|cuda/3.0
|''default's short name unchanged''
|-
|cuda/cuda-3.1
|cuda/3.1
|
|-
|debuggers/ddd-3.3.12
|ddd/3.3.12
|
|-
|debuggers/gdb-7.1
|gdb/7.1
|
|-
|editors/nano/2.2.4
|nano/2.2.4
|
|-
|emacs/emacs-23.1
|emacs/23.1.1
|''short name unchanged''
|-
|gcc/gcc-4.4.0
|gcc/4.4.0
|''short name unchanged''
|-
|graphics/ncview
|ncview/1.93
|
|-
|graphics/graphics
|grace/5.1.22
|
|-
|
|gnuplot/4.2.6
|
|-
|svn/svn165
|svn/1.6.5
|''short name unchanged''
|-
|visualization/paraview
|paraview/3.8
|
|-
|amber10/amber10
|amber/10.0.30
|
|-
|gamess/gamess
|gamess/May2209  
|''default's short name unchanged''
|}

==modulefind - Finding modules by name==

The <tt>module avail</tt> command will only show you modules whose names start with the argument that you give it, and will alsi return modules that you cannot load due to conflicts with already loaded modules.

A little SciNet utility called <tt>modulefind</tt> (one word) can do that. It will list all installed modules which contain the arguments, and will determine whether those modules have
been loaded, could be loaded, cannot because of conflicts with
already loaded modules, or have unresolved dependencies
(i.e. for which other modules need to be loaded first). This is especially useful in cases like the "boost" libraries, whose module names are cxxlibraries/boost/1.47.0-gcc and cxxlibraries/boost/1.47.0-gcc, for the gcc and intel compiler, respectively. <tt>modulefind boost</tt> will find those, whereas <tt>module avail boost</tt> will not.

Note that just 'modulefind' will list all top-level modules.

== Making your own modules ==

How to make your own modules (e.g. for local installations or to access optional perl modules, ...), is possible and described on the [[Installing your own modules]] page.

== Deprecated modules ==

Some older software modules for which newer versions exist, get deprecated, which means they do not get maintained. Since deprecated modules should only be needed in rare exceptional cases, they are not listed by the <tt>module avail</tt> command. However, if you have a piece of legacy code that really depends on a deprecated version of a library (and we urge you to check that it does not work with newer versions!), then you can load a deprecated version by <pre>module load use.deprecated [deprecated-module-name]</pre>

Currently (Oct 5,2010), the following modules are deprecated on the GPC:
<pre>
gcc/gcc-4.3.2 hdf5/184-v16-serial intel/intel-v11.1.046 openmpi/1.3.3-intel-v11.0-ofed
hdf5/183-v16-openmpi hdf5/184-v18-intelmpi intelmpi/impi-3.2.1.009 openmpi/1.3.2-intel-v11.0-ofed.orig
hdf5/183-v18-openmpi hdf5/184-v18-openmpi intelmpi/impi-3.2.2.006 pgplot/5.2.2-gcc.old
hdf5/184-v16-intelmpi hdf5/184-v18-serial intelmpi/impi-4.0.0.013 pgplot/5.2.2-intel.old
hdf5/184-v16-openmpi intel/intel-v11.0.081 intelmpi/impi-4.0.0.025
</pre>

On the TCS, currently (Oct 5,2010) the only deprecated module is:
<pre>
ncl/5.1.1old
</pre>

Before using any of these deprecated modules, make sure that there is not a regular module that satisfies your needs, likely by a ''very similar name''.

== Commercial software ==

Apart from the compilers on our systems and the ddt parallel debugger, we generally do not provide licensed application software, e.g., no Gaussian, IDL, Matlab, etc.
See the [https://support.scinet.utoronto.ca/wiki/index.php/FAQ#How_can_I_run_Matlab_.2F_IDL_.2F_Gaussian_.2F_my_favourite_commercial_software_at_SciNet.3F FAQ].

== Other software and libraries ==

If you want to use a piece of software or a library that is not on the list, you can in principle install it yourself in you <tt>/home</tt> directory.
Note however that building libraries and software from source often uses a lot of files. To avoid running out of disk space, building software is therefore best done from the <tt>/scratch</tt>, from which
you can copy/install only the libraries, header files and binaries to your <tt>/home</tt> directory.

If you suspect that a particular piece of software or a library would be of use to other users of SciNet as well, contact us, and we will consider adding it to the system.

== Software lists ==
=== ARC/GPU Software ===

The CPUs in the GPU nodes of the ARC cluster are of the same kind as those of the GPC, so all modules available on the GPC are available on the GPU nodes with a CentOS 6 image. This means that the different cuda variants that are available as modules, can be loaded on those GPC nodes as well, although they are of little use on that system.

=== GPC Software ===

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Software
!{{Hl2}}| Versions
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|Intel Compiler
|12.1.3*, 12.1.5, 13.1.1, 14.0.1
| includes MKL library, which includes BLAS, LAPACK, FFT, ...
| <tt>icpc,icc,ifort</tt>
| <tt>intel</tt>
|-
| GCC Compiler
| 4.4.6, 4.6.1*, 4.7.0, 4.7.2, 4.8.1, 4.9.0
|
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Cuda
| 3.2, 4.0, 4.1*, 4.2, 5.0, 5.5
| NVIDIA's extension to C for GPGPU programming
| <tt>nvcc</tt>
| <tt>cuda</tt>
|-
| PGI Compiler
| 12.5
| supports OpenACC and CUDA Fortran
| <tt>pgcc,pgcpp,pgfortran</tt>
| <tt>pgi</tt>
|-
| IntelMPI
| 4.0.2, 4.1.2
| MPICH2 based MPI
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>intelmpi</tt>
|-
| OpenMPI
| 1.4.4*, 1.5.4, 1.6.4
|
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>openmpi</tt>
|-
| UPC
| 2.12.2
| Berkley Unified Parallel C Implementation
| <tt>upcc</tt>
| <tt>upc</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Editors'''''
|-
| Nano
| 2.2.4
| Nano's another editor
| <tt>nano</tt>
| <tt>nano</tt>
|-
| Emacs
| 23.1.1
| New version of popular text editor
| <tt>emacs</tt>
| <tt>emacs</tt>
|-
| XEmacs
| 21.4.22
| XEmacs editor
| <tt>xemacs</tt>
| <tt>xemacs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Development tools'''''
|-
| Autoconf
| 2.68
|
| <tt>autoconf, ...</tt>
| <tt>autoconf</tt>
|-
| Automake
| 1.11.2
|
| <tt>aclocal, automake</tt>
| <tt>automake</tt>
|-
| CMake
| 2.8.6
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
| Scons
| 2.0
| Software construction tool
| <tt>scons</tt>
| <tt>scons</tt>
|-
| Git
| 1.7.1
| Revision control system
| <tt>git,gitk</tt>
| <tt>git</tt>
|-
| Intel tools
| 2011
| Intel Code Analysis Tools
| Vtune Amplifier XE, Inspector XE
| <tt>inteltools</tt>
|-
| Mercurial
| 1.8.2
| Version control system (part of the python module!)
| <tt>hg</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug and performance tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1, 4.2.1*
| Allinea's Distributed Debugging Tool, + MAP MPI Profiler
| <tt>ddt, map</tt>
| <tt>ddt</tt>
|-
| DDD
| 3.3.12
| Data Display Debugger
| <tt>ddd</tt>
| <tt>ddd</tt>
|-
| GDB
| 7.3.1
| GNU debugger (the intel idbc debugger is available by default)
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| MPE2
| 2.4.5
| Multi-Processing Environment with intel + OpenMPI
| <tt>mpecc, mpefc, jumpshot</tt>
| <tt>mpe</tt>
|-
| [[Introduction_To_Performance#OpenSpeedShop_.28profiling.2C_MPI_tracing:_GPC.29 | OpenSpeedShop]]
| 2.0.1
| sampling and MPI tracing
| <tt>openss, ...</tt>
| <tt>openspeedshop</tt>
|-
| [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Scalasca]]
| 1.2
| SCalable performance Analysis of LArge SCale Applications (Compiled with OpenMPI)
| <tt>scalasca</tt>
| <tt>scalasca</tt>
|-
| [http://ipm-hpc.sourceforge.net IPM]
| 0.983
| Integrated Performance Monitors http://ipm-hpc.sourceforge.net/]
| <tt>ipm, ipm_parse, ploticus,...</tt>
| <tt>ipm</tt>
|-
| [[Performance_And_Debugging_Tools:_GPC#Valgrind | Valgrind]]
| 3.6.1
| Memory checking utility
| <tt>valgrind,cachegrind</tt>
| <tt>valgrind</tt>
|-
| Padb
| 3.2
| examine and debug parallel programs
| <tt>padb</tt>
| <tt>padb</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Visualization tools'''''
|-
| Grace
| 5.1.22
| Plotting utility
| <tt>xmgrace</tt>
| <tt>grace</tt>
|-
| Gnuplot
| 4.2.6
| Plotting utility Requires 'extras' module if used on compute nodes.
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| [[ Using_Paraview | ParaView ]]
| 3.12.0
| Scientific visualization, server only
| <tt>pvserver,pvbatch,pvpython</tt>
| <tt>paraview</tt>
|-
| VMD
| 1.9
| Visualization and analysis utility
| <tt>vmd</tt>
| <tt>vmd</tt>
|-
| NCL/NCARG
| 6.0.0
| NCARG graphics and ncl utilities
| <tt>ncl</tt>
| <tt>ncl</tt>
|-
| ROOT
| 5.30.00
| ROOT Analysis Framework from CERN
| <tt>root</tt>
| <tt>ROOT</tt>
|-
| ImageMagick
| 6.6.7
| Image manipulation tools
| <tt>convert,animate,composite,...</tt>
| <tt>ImageMagick</tt>
|-
| PGPLOT
| 5.2.2
| Graphics subroutine library
| <tt>libcpgplot,libpgplot,libtkpgplot</tt>
| <tt>pgplot</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf</tt>
|-
| Ncview
| 2.1.1
| Visualization for NetCDF files
| <tt>ncview</tt>
| <tt>ncview</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, etc.</tt>
| <tt>nco</tt>
|-
| CDO
| 1.5.1
| Climate Data Operators
| <tt>cdo</tt>
| <tt>cdo</tt>
|-
| UDUNITS
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| HDF4
| 4.2.6
| Scientific data storage and retrieval
| <tt>h4fc,hdiff,...,libdf,libsz</tt>
| <tt>hdf4</tt>
|-
| [[Hdf5 | HDF5]]
| 1.8.7-v18*
| Scientific data storage and retrieval, parallel I/O
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5</tt>
|-
| [http://www.arg0.net/encfs EncFS ]
| 1.74
| EncFS provides an encrypted filesystem in user-space, (works ONLY on gpc01..04)
| <tt>encfs</tt>
| <tt>encfs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| [[amber|AMBER 10]]
| Amber 10 + Amber Tools 1.3
| Amber Molecular Dynamics Package
| <tt>sander, sander.MPI</tt>
| <tt>amber</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| [[gamess|GAMESS (US)]]
| August 18, 2011 R1
| General Atomic and Molecular Electronic Structure System
| <tt>rungms</tt>
| <tt>gamess</tt>
|-
| [[gromacs|GROMACS]]
| 4.5.5, 4.5.7, 4.6.2
| GROMACS molecular dynamics, single precision, MPI
| <tt>grompp, mdrun</tt>
| <tt>gromacs</tt>
|-
| [[namd|NAMD]]
| 2.8
| NAMD - Scalable Molecular Dynamics
| <tt>namdmpiexec, namd2</tt>
| <tt>namd</tt>
|-
| [[nwchem|NWChem]]
| 6.0
| NWChem Quantum Chemistry
| <tt>nwchem</tt>
| <tt>nwchem</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 4.3.2, 5.0.3
| Quantum Chemistry
| <tt>pw.x, ...</tt>
| <tt>espresso</tt>
|-
| [http://blast.ncbi.nlm.nih.gov BLAST]
| 2.2.23+
| Basic Local Alignment Search Tool
| <tt>blastn,blastp,blastx,psiblast,tblastn...</tt>
| <tt>blast</tt>
|-
|-
| [http://denovoassembler.sourceforge.net RAY]
| 2.1.0 (small k-mer)
| Parallel de novo genome assemblies
| <tt>Ray</tt>
| <tt>ray</tt>
|-
| [[cpmd|CPMD]]
| 3.13.2
| Carr-Parinello molecular dynamics, MPI
| <tt>cpmd.x</tt>
| <tt>cpmd</tt>
|-
| [[R Statistical Package|R]]
| 2.13.1
| statistical computing
| <tt>R</tt>
| <tt>R</tt>
|-
| Octave
| 3.4.3
| Matlab-like environment
| <tt>octave</tt>
| <tt>octave</tt>
|-
|-
| [http://www.openfoam.org OpenFOAM ]
| 2.3.0
| Open Source CFD Package
| <tt>*foam</tt>
| <tt>openfoam</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| [http://www.mcs.anl.gov/petsc/petsc-as/ PETSc ]
| 3.1*
| Portable, Extensible Toolkit for Scientific Computation (PETSc)
| <tt>libpetsc, etc.. </tt>
| <tt>petsc</tt>
|-
| BOOST
| 1.47.0
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| [[Armadillo C++ linear algebra library | Armadillo]]
| 3.910.0
| C++ armadillo libraries (implement Matlab-like syntax)
| <tt>armadillo</tt>
| <tt>armadillo</tt>
|-
| [http://www.tacc.utexas.edu/tacc-projects/gotoblas2/ GotoBLAS]
| 1.13
| Optimized BLAS implementation
| <tt>libgoto2</tt>
| <tt>gotoblas</tt>
|-
| GSL
| 1.13*, 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| FFTW
| 3.3
| fast Fourier transform library
''Be careful in combining fftw3 and MKL: you need to link fftw3 first, with'' <tt>-L${SCINET_FFTW_LIB} -lfftw3</tt>, then link MKL
| <tt>libfftw3</tt>
| <tt>fftw</tt>
|-
| LAPACK
|
| Provided by the Intel MKL library
| See http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/
| <tt>intel</tt>
|-
| [http://freshmeat.net/projects/rlog RLog ]
| 1.4
| RLog provides a flexible message logging facility for C++ programs and libraries.
| <tt>librlog</tt>
| <tt>cxxlibraries/rlog</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[GNU Parallel]]
| 2012-10-22
| execute commands in parallel
| <tt>parallel</tt>
| <tt>gnu-parallel</tt>
|-
| [[Python]]
| 2.7.2
| Python programming language
| <tt>python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.3.4
| Python programming language. Modules included : numpy 1.8.1 , scipy 0.14.0 , matlotlib 1.3.1 , ipython 1.2.1 , cython 0.20.1 , h5py-2.3.0 , tables-3.1.1 , netCDF4-1.1.0 , astropy-0.3.2 , scikit_learn-0.15.0b1
| <tt>python</tt>
| <tt>python</tt>
|-
| Ruby
| 1.9.1
| Ruby programming language
| <tt>ruby</tt>
| <tt>ruby</tt>
|-
| Java
| 1.6.0
| IBM's Java JRE ad SDK
| <tt>java</tt>, <tt>javac</tt>
| <tt>java</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| Xlibraries
|
| A collection of X graphics libraries and tools
| <tt>xterm</tt>, <tt>xpdf</tt>, ...
| <tt>Xlibraries</tt>
|-
| Extras
|
| A collection of standard linux and home-grown tools
| <tt>bc</tt>, <tt>screen</tt>, <tt>xxdiff</tt>, <tt>modulefind</tt>, <tt>ish</tt>, ...
| <tt>extras</tt>
|}

''* Several versions of this module are installed; listed is the default version.''

=== TCS Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM compilers
|10.1(c/c++) 12.1(fortran)
| See [[TCS Quickstart]]
| <tt>xlc,xlC,xlf,xlc_r,xlC_r,xlf_r</tt>
| ''standard available''
|-
|IBM MPI library
|
| See [[TCS Quickstart]]
| <tt>mpcc,mpCC,mpxlf,mpcc_r,mpCC_r,mpxlf_r</tt>
| ''standard available''
|-
| UPC
| 1.2
| Unified Parallel C
| <tt>xlupc</tt>
| <tt>upc</tt>
|-
|IBM fortran compiler
|13.1, 14.1
| newer version
| <tt>xlf,xlf_r</tt>
| xlf/13.1
|-
|IBM c/c++ compilers
|11.1, 12.1
| new versions
| <tt>xlc,xlC,xlc_r,xlC_r</tt>
| vacpp
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| MPE2
| 1.0.6
| Performance Visualization for Parallel Programs
| <tt>libmpe</tt>
| <tt>mpe</tt>
|-
| Scalasca
| 1.2
| SCalable performance Analysis of LArge SCale Applications
| <tt>scalasca, ...</tt>
| <tt>scalasca</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF4
| 4.2.5
| Scientific data storage and retrieval
| <tt>h4fc, hdiff, ..., libdf, libsz</tt>
| <tt>hdf4</tt>
|-
| HDF5
|
| Scientific data storage and retrieval, parallel I/O Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libhdf5</tt>
| <tt>extras</tt>
|-
| NetCDF + ncview
| 4.0.1*
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf, ncview</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.1.1*
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 3.9.6*
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| FFTW
| 3.2.2
| Fast Fourier transform library Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi,libfftw3</tt>
| <tt>extras</tt>
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| LAPACK+SCALAPACK
| 3.4.2+2.0.2
| Linear algebra package. Note that essl, which comes with the ibm compilers contains a large part of lapack as well.
| <tt>liblapack,libscalapack,libblacs</tt>
| <tt>lapack</tt>
|-
| PetSc
| 3.2
| Portable, Extensible Toolkit for Scientific Computation. With external packages mumps, chaco, hypre, parmetis, prometheus, plapack, superlu, sprng.
| <tt>libpetsc,...</tt>
| <tt>petsc</tt>
|-
| extras
|
| Adds paths to a fuller set of libraries to your user environment compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi, libfftw3, libhdf5, liblapack, ...</tt>
| <tt>extras</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gmake
| 3.82
| GNU's make. Replaces AIX make or gmake 3.80.
| <tt>make</tt>, <tt>gmake</tt>
| <tt>gmake</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| NCL
| 5.1.1
| NCAR Command Language
| <tt>ncl, libncl, ...</tt>
| <tt>ncl</tt>
|-
|}
''* Several versions of this module are installed; listed is the default version.''

=== P7 Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1, 13.1
|See [[P7 Linux Cluster]]
|<tt>xlf,xlf_r,xlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1, 11.1
|See [[P7 Linux Cluster]]
|<tt>xlc,xlC,xlc_r,xlC_r,...</tt>
|vacpp
|-
| Binutils
| 2.23.2
|
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
|IBM MPI library
|5.2.2
|IBM's Parallel Environment
|<tt>mpcc,mpCC,mpfort,mpiexec</tt>
|pe
|-
| GCC Compiler
| 4.6.1 , 4.8.1
| GNU Compiler Collection
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Java
| 7.0
| IBM Java 1.7 implementation
| <tt>javac</tt>
| <tt>jdk</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF5
| 1.8.7
| Scientific data storage and retrieval, parallel I/O
| <tt>libhdf5</tt>
| <tt>hdf5</tt>
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.2.0
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.7.5
| Python programming language. Modules included : numpy-1.8.0 , scipy-0.13.2 , matplotlib-1.3.1 , pyfits-3.2 , h5py-2.2.1
| <tt>python</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gnuplot
| 4.6.1
| command-driven interactive function and data plotting program
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| udunits
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| extras
|
| Adds paths to a fuller set of applications and libraries to your user environment
| <tt>bindlaunch, ...</tt>
| <tt>extras</tt>
|}

=Manuals=
{{:Manuals}}

Oldwiki.scinet.utoronto.ca:System Alerts

2014-08-28T14:30:43Z

Brelier: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2014-08-27T19:46:06Z

Brelier: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up75.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|}

Wed Aug 27 15:44:00 EDT 2014: Issue with a node board on the production system : full system jobs cannot run until the node is fixed.

On August 21, 2014, at 00:30am, there will be a 15 minute period during which maintenance on our gateway switch must be conducted. As a result you may not be able to contact the datacentre. No jobs will be affected by this.

Note: As a precaution, emails by the Moab/Torque scheduler have been disabled because of a potential security vulnerability since Jan 24th 2014.

Last updated on Wed Aug 20 11:19:21 EDT 2014

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2014-08-27T19:45:46Z

Brelier: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[up75.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|}

Wed Aug 27 15:44:00 EDT 2014: Issue with a node board on the production system : full system jobs cannot run until the node is fixed.

On August 21, 2014, at 00:30am, there will be a 15 minute period during which maintenance on our gateway switch must be conducted. As a result you may not be able to contact the datacentre. No jobs will be affected by this.

Note: As a precaution, emails by the Moab/Torque scheduler have been disabled because of a potential security vulnerability since Jan 24th 2014.

Last updated on Wed Aug 20 11:19:21 EDT 2014

([[Previous_messages:|Previous messages]])

BGQ

2014-08-19T18:45:30Z

Brelier: /* Steps ( Job dependency) */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>
=== Steps ( Job dependency) ===
LoadLeveler has a lot of advanced features to control job submission and execution. One of these features is called steps. This feature allows a series of jobs to be submitted using one script with dependencies defined between the jobs. What this allows is for a series of jobs to be run sequentially, waiting for the previous job, called a step, to be finished before the next job is started. The following example uses the same LoadLeveler script as previously shown, however the #@ step_name and #@ dependency directives are used to rerun the same case three times in a row, waiting until each job is finished to start the next.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step1
# @ queue
# Launch the first step :
if [ $LOADL_STEP_NAME = "step1" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step2
# @ dependency = step1 == 0
# @ queue
# Launch the second step if the first one has returned 0 (done successfully) :
if [ $LOADL_STEP_NAME = "step2" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step3
# @ dependency = step2 == 0
# @ queue
# Launch the third step if the second one has returned 0 (done successfully) :
if [ $LOADL_STEP_NAME = "step3" ]; then
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
fi

</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

debugjob session with an executable implicitly calls runjob with 1 mpi task :
<pre>
debugjob -i
**********************************************************
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and
LL14040718574824 for 30 minutes with 64 NODES (1024 cores).
IMPLICIT MODE: running an executable implicitly calls runjob
with 1 mpi task
Exit shell when finished.
**********************************************************
</pre>

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-08-19T17:54:00Z

Brelier: /* Steps ( Job dependency) */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>
=== Steps ( Job dependency) ===
LoadLeveler has a lot of advanced features to control job submission and execution. One of these features is called steps. This feature allows a series of jobs to be submitted using one script with dependencies defined between the jobs. What this allows is for a series of jobs to be run sequentially, waiting for the previous job, called a step, to be finished before the next job is started. The following example uses the same LoadLeveler script as previously shown, however the #@ step_name and #@ dependency directives are used to rerun the same case three times in a row, waiting until each job is finished to start the next.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step1
# @ queue
# Launch the first step :
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags

# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step2
# @ dependency = step1 == 0
# @ queue
# Launch the second step if the first one has returned 0 (done successfully) :
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags

# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ step_name = step3
# @ dependency = step2 == 0
# @ queue
# Launch the third step if the second one has returned 0 (done successfully) :
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags

</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

debugjob session with an executable implicitly calls runjob with 1 mpi task :
<pre>
debugjob -i
**********************************************************
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and
LL14040718574824 for 30 minutes with 64 NODES (1024 cores).
IMPLICIT MODE: running an executable implicitly calls runjob
with 1 mpi task
Exit shell when finished.
**********************************************************
</pre>

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-08-19T17:41:22Z

Brelier: /* Steps ( Job dependency) */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>
=== Steps ( Job dependency) ===
LoadLeveler has a lot of advanced features to control job submission and execution. One of these features is called steps. This feature allows a series of jobs to be submitted using one script with dependencies defined between the jobs. What this allows is for a series of jobs to be run sequentially, waiting for the previous job, called a step, to be finished before the next job is started. The following example uses the same LoadLeveler script as previously shown, however the #@ step_name and #@ dependency directives are used to rerun the same case three times in a row, waiting until each job is finished to start the next.

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

debugjob session with an executable implicitly calls runjob with 1 mpi task :
<pre>
debugjob -i
**********************************************************
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and
LL14040718574824 for 30 minutes with 64 NODES (1024 cores).
IMPLICIT MODE: running an executable implicitly calls runjob
with 1 mpi task
Exit shell when finished.
**********************************************************
</pre>

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-08-19T17:40:42Z

Brelier:

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>
=== Steps ( Job dependency) ===

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

debugjob session with an executable implicitly calls runjob with 1 mpi task :
<pre>
debugjob -i
**********************************************************
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and
LL14040718574824 for 30 minutes with 64 NODES (1024 cores).
IMPLICIT MODE: running an executable implicitly calls runjob
with 1 mpi task
Exit shell when finished.
**********************************************************
</pre>

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

GPC Quickstart

2014-07-11T19:16:35Z

Brelier: /* MPI */

{{Infobox Computer
|image=[[Image:University_of_Tor_79284gm-a.jpg|center|300px|thumb]]
|name=General Purpose Cluster (GPC)
|installed=June 2009
|operatingsystem= Linux
|loginnode= gpc01..gpc04 (from <tt>login.scinet</tt>)
|nnodes=3864 (30,912 cores)
|rampernode=16 Gb
|corespernode=8 (16 threads)
|interconnect=840 nodes 1:1 DDR, 3024 nodes 5:1 QDR
|vendorcompilers=icc (C) ifort (fortran) icpc (C++)
|queuetype=[[Moab | Moab/Torque]]
}}

__TOC__

===Specifications===
The General Purpose Cluster is an extremely large cluster (ranked [http://www.top500.org/list/2009/06/100 16th] in the world at its inception, and fastest in Canada) and is where most computations are to be done at SciNet. It is an IBM iDataPlex cluster based on Intel's Nehalem architecture (one of the [http://www.hpcwire.com/features/HPC-Vendors-Jump-On-Nehalem-42360237.html first in the world] to make use of the new chips). The GPC consists of 3,780 nodes (IBM iDataPlex DX360M2) with a total of 30,912 cores (Intel Xeon E5540) at 2.53GHz, with 16GB RAM per node (2GB per core). Approximately one quarter of the cluster is interconnected with non-blocking DDR InfiniBand while the rest of the nodes are connected with 5:1 blocked QDR InfiniBand. The compute nodes are accessed through a queuing system that allows jobs with a minimum of 15minutes and maximum wall time of 48 hours .

===Login===

First login via [[Ssh | ssh]] with your SciNet account at <tt>login.scinet.utoronto.ca</tt>, and from there you can proceed to the Development nodes (<tt>gpc01</tt>, <tt>gpc02</tt>, <tt>gpc03</tt>, or <tt>gpc04</tt>) to compile/test your code.

===Compile/Devel Nodes===

From a scinet login node you can ssh to <tt>gpc01</tt>..<tt>gpc04</tt>. You may also just use <tt>/scinet/gpc/bin/gpcdev</tt> to take directly to the dev node with lowest cpu load (reassessed every 5 minutes).
<pre>
% /scinet/gpc/bin/gpcdev -h
Usage: gpcdev [usual ssh options]
Example: gpcdev -X
redirected to dev node with lowest cpu load
</pre>

These nodes have the same hardware configuration as most of the compute nodes except with more memory -- 8 Nehalem processing cores with 36GB RAM and QDR Infiniband. You can compile and test your codes on these nodes. To interactively test on more than 8 processors, you can submit an [[GPC_Quickstart#Submitting_an_Interactive_.28Debug.29_Job | interactive job request]].

Your [[Storage_Quickstart | home directory]] is in <tt>$HOME</tt> (currently <tt>/home/g/group/USER</tt> but this can change, so it's best to use the environment variable $HOME or <tt>~</tt> in scripts). You have 10GB there that is backed up. '''Your home directory cannot be written to by the compute nodes!''' Thus, to run jobs, you'll use the <tt>$SCRATCH</tt> directory (currently <tt>/scratch/g/group/USER</tt> but again, use the environment variable). Here, there is a large amount of disk space, but it is not backed up. Thus it makes sense to keep your codes in /home, compile there, and then run them in the /scratch directory.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including any of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

Note that to use even the gcc compilers you will have to do
<pre>
$ module load gcc
</pre>
but in fact you probably should use the intel compilers installed on this system as they often produce faster executables (and occasionally, much faster.)

A list of the installed software is available in [[Software_and_Libraries | Software & Libraries]] and can
be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load intel
</pre>
To unload a module
<pre>
$ module unload intel
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands should got in your submission scripts to make sure you
are using the correct packages. It is possible to load them in your .bashrc files as well, but this is generally not recommended (see [[Important .bashrc guidelines]]), especially if you routinely have to flip back and forth between modules.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' affect other shell environments; in particular, a queued job that is running is unaffected by you interactively loading a module, and conversely you loading a module at the prompt and then submitting a job does not ensure that the module is loaded when the job runs. To ensure that a module is loaded when a job runs, be sure to put your <tt>module load</tt> command in your job submission script.

===Compilers===

The intel compilers are icc/icpc/ifort for C/C++/Fortran, and are available with the default module "intel". The intel compilers are recommended over the GNU compilers. Documentation about icpc is available at
http://software.intel.com/en-us/articles/intel-software-technical-documentation/. The Intel compilers accept many of the options that the GNU compilers accept, but tend to produce faster programs on our system. If, for some reason, you really need the GNU compilers, the latest version of the GNU compiler collection (currently 4.4.0) is available by loading the "gcc" module, with gcc/g++/gfortran for C/C++/Fortran. Note that f77/g77 is not supported.

To ensure that the intel compilers are in your <tt>PATH</tt> and their libraries are in your <tt>LD_LIBRARY_PATH</tt>, use the command

<pre>
$ module load intel
</pre>

Optimize your code for the GPC machine using of at least the following compiler flags:
<pre>
-O3 -xHost
</pre>
(or <tt>-O3 -march=native</tt> for the GNU compilers).

*If your program uses openmp, add <tt>-openmp</tt> (<tt>-fopenmp</tt> for GNU compilers).
*If you get the warning <tt>feupdatreenv is not implemented</tt>, add -limf to the link line.
*If you need to link in the MKL libraries, you are well advised to use the Intel(R) Math Kernel Library Link Line Advisor: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ for help in devising the list of libraries to link with your code. '''Note that this give the link line for the command prompt. When using this in Makefiles, replace $MKLPATH by ${MKLPATH}.'''
*More questions about compiling? See the [[FAQ#Compiling_your_Code|FAQ]].

===Debuggers===

* '''ddt''' - Allinea's graphical parallel debugger, in the <tt>ddt</tt> module. Highly recommended!
* '''gdb''' - The GNU Debugger, available in the <tt>gdb</tt> module.
* '''idbc/idb''' - The intel debuggers, part of the <tt>intel</tt> module(s).
* '''ddd''' - A graphical debuggerThe GNU Debugger, available in the <tt>ddd</tt> module.

Note that to debug code, you have to give the <pre>-g</pre> flags to the compiler. The intel compiler needs the additional option <tt>-debug parallel</tt> to debug threaded/OpenMP code.

===[[ GPC_MPI_Versions | MPI]]===

SciNet currently provides multiple MPI libraries for the GPC; [http://www.open-mpi.org/ OpenMPI], and [http://software.intel.com/en-us/intel-mpi-library/ IntelMPI]. We currently recommend OpenMPI as the default, as it quite reliably demonstrates good performance on the infiniband network (and did so too on the ethernet network). For full details and options see the complete [[ GPC_MPI_Versions | '''MPI''']] section.

The MPI libraries are compiled with both the gnu compiler suite and the intel compiler suite. To use (for instance) the intel-compiled OpenMPI libraries, which we recommend as the default (and use for most of our examples here), use

<pre>
$ module load intel openmpi
</pre>

in your job submission scripts and on the command-line before compiling. Putting these in your <tt>.bashrc</tt> is no longer recommended (since Oct 10, 2013).

Other combinations behave similarly.

The MPI libraries define the wrappers mpicc/mpicxx/mpif90/mpif77 as wrappers around the appropriate compilers, which ensure the appropriate include and library directories are used in the compilation and linking steps.

We currently recommend the Intel + OpenMPI combination. However, if you require the GNU compilers as well as MPI, you will want to find the most recent openmpi module available with `gcc' in the version name. This will enable development and runtime with gcc/g++/gfortran and OpenMPI.


For mixed OpenMP/MPI code using Intel MPI, add the compilation flag -mt_mpi for full thread-safety (no such flag is necessary for OpenMPI).

===Submitting A Batch Job===

The SciNet machines are shared systems, and jobs that are to run on them are submitted to a queue; the
[[Moab | scheduler]] then orders the jobs in order to make the best use of the machine, and has them launched
when resources become availble. The intervention of the scheduler can mean that the jobs aren't
quite run in a first-in first-out order.

The scheduler will have the job run on one or more of the compute nodes of the GPC. It is important to realize that '''on compute nodes, your home directory is read-only'''. You have to run your jobs from the $SCRATCH directory instead. See [https://support.scinet.utoronto.ca/wiki/index.php/Data_Management Data Management ] for more details on the file systems at SciNet.

The maximum [[wallclock time]] for a job in the queue is 48 hours; computations that will take longer than
this must be broken into 48-hour chunks and run as several jobs. Also a minimum job length of 15 minutes is enforced and shorter jobs
should be batched together. The usual way to do this is with [[checkpoints]],
writing out the complete state of the computation every so often in such a way that a job can be restarted from
this state information and continue on from where it left off. Generating [[checkpoints]] is a good idea anyway,
as in the unlikely event of a hardware failure during your run, it allows you to restart without having lost much work.

There are limits to how many jobs you can submit. If your group has a default account, up to 32 nodes at a time for 48 hours per job on the GPC cluster are allowed to be queued. This is a total limit, e.g., you could request 64 nodes for 24 hours. Jobs of users with an LRAC or NRAC allocation will run at a higher priority than others while their resources last. Because of the group-based allocation, it is conceivable that your jobs won't run if your colleagues have already exhausted your group's limits.

Note that scheduling big jobs greatly affects the queuer and other users, so you have to talk to us first to run massively parallel jobs (> 2048 cores). We will help make sure that your jobs start and run efficiently.

If your job should run in fewer than 48 hours, specify that in your script -- your job
will start sooner. (It's easier for the [[Moab | scheduler]] to fit in a short job than a long job). On the downside, the
job will be killed automatically by the queue manager software at the end of the specified [[wallclock time]], so if you
guess wrong you might lose some work. So the standard procedure is to estimate how long your job will take and
add 10% or so.

You interact with the queuing system through the queue/resource manager, [[Moab | Moab]] and [[Moab | Torque]]. To see all the jobs in the queue use
<pre>
$ showq
</pre>

To submit your own job, you must write a script which describes the job and how it is to be run (a sample script [[GPC_Quickstart#Submission_Script | follows]]) and submit it to the queue, using the command
<pre>
$ qsub [SCRIPT-FILE-NAME]
</pre>
where you will replace <tt>[SCRIPT-FILE-NAME]</tt> with the file containing the submission script. This will return a job ID, for example 31415, which is used to identify the jobs. Information about a queued job can be found using
<pre>
$ checkjob [JOB-ID]
</pre>
and jobs can be canceled with the command
<pre>
$ canceljob [JOB-ID]
</pre>

Again, these commands have many options, which can be read about on their man pages.

Much more information on the queueing system is available on our [[Moab | queue]] page.

====Batch Submission Script: MPI====

A sample submission script is shown below for an mpi job with the <tt> #PBS </tt> directives at the top and the rest being
what will be executed on the compute node.

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC
#
#PBS -l nodes=2:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel openmpi

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -np 16 ./a.out
</source>

The lines that begin <tt>#PBS</tt> are commands that are parsed and interpreted by qsub at submission time, and control administrative things about your job. In this example, the script above requests two nodes, using 8 processors per node, for a [[wallclock time]] of one hour. (The resources required by the job are listed on the <tt>#PBS -l</tt> line.) Other options can be given in other <tt>#PBS</tt> lines, such as <tt>#PBS -N</tt>, which sets the name of the job.

The rest of the script is run as a bash script at run time. A bash shell on the first node of the two nodes that are requested executes these commands as a normal bash script, just as if you had run this as a shell script from the terminal. The only difference is that PBS sets certain environment variables that you can use in the script. <tt>$PBS_O_WORKDIR</tt> is set to be the directory that the command was 'submitted' from - eg, <tt>$SCRATCH/SOMEDIRECTORY</tt>. The script then uses the <tt>mpirun</tt> command to launch the job.



====Submitting Collections of Serial Jobs====

You cannot run purely serial jobs on the GPC (or any of SciNet's systems), as this would mean only one core out of 8 is used. If you have serial jobs, you have to bunch them together.
SciNet-approved methods for running collections of serial jobs can be found on the [[User_Serial|serial run wiki page]].

====Batch Submission Script: OpenMP====

For running OpenMP jobs, the procedure is similar as for MPI jobs:

<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (OpenMP)
#
#PBS -l nodes=1:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

export OMP_NUM_THREADS=8
./a.out
</source>

Note that [[Introduction_To_Performance#Throughput | in some circumstances]] it can be more efficient to run (say) two jobs each running on four threads than one job running on eight threads. In that case you can use the same `ampersand-and-wait' technique outlined for serial jobs (see [[User_Serial|serial run wiki page]]) for less-than-eight-core OpenMP jobs.

====Hybrid MPI/OpenMP jobs====

'''Using Intel MPI'''

Here is how to run hybrid codes using intelmpi::

http://software.intel.com/en-us/articles/hybrid-applications-intelmpi-openmp/

Make sure you compile with the -mt_mpi option to the compilers to use the thread safe libraries.
Set the environment variable I_MPI_PIN_DOMAIN:
<pre>
$ export I_MPI_PIN_DOMAIN=omp
</pre>
This will set the process pinning domain size to be equal to OMP_NUM_THREADS (which you should set to the desired number of threads per mpi process). Therefore, each MPI process can create $OMP_NUM_THREADS number of children threads for running within the corresponding domain. If OMP_NUM_THREADS is not set, each node is treated as a separate domain (which will allow as many threads per MPI processes as there are cores).

In addition, when invoking mpirun, you should add the argument "-ppn X", where X is the number of MPI processes per node.
For example:
<pre>
$ mpirun -ppn 2 -np 8 [executable]
</pre>
would start 2 mpi processes of <tt>[executable]</tt> per node for a total of 8 processes, so mpirun will try to run mpi processes on 4 nodes
(OMP_NUM_THREADS is then probably best set at 4).
Your job script should still ask for these 4 nodes with the line
<source lang="bash">
#PBS -l nodes=4:ppn=8,walltime=....
</source>
(<tt>ppn=8</tt> is not a mistake here; the ppn parameter has a different meaning for PBS and for mpirun)

''The ppn parameter to ''mpirun'' is very important! Without it, eight mpi jobs would get bunched on the first node in this example, leaving 3 nodes unused.''

NOTE: In order to pin OpenMP threads inside the domain, use the corresponding OpenMP feature by setting the KMP_AFFINITY environment variable, see [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/fortran/lin/compiler_f/optaps/common/optaps_openmp_thread_affinity.htm#KMP_AFFINITY_Environment_Variable|Intel's Compiler User and Reference Guide].

The IntelMPI manual is referenced on the front page of our wiki:

http://software.intel.com/sites/products/documentation/hpc/mpi/linux/reference_manual.pdf

For the above example of a total of 8 processes on 4 nodes, you could use the following script:
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (hybrid job)
#
#PBS -l nodes=4:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel intelmpi

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# SET THE NUMBER OF THREADS PER PROCESS:
export OMP_NUM_THREADS=4

# PIN THE MPI DOMAINS ACCORDING TO OMP
export I_MPI_PIN_DOMAIN=omp

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -ppn 2 -np 8 ./a.out
</source>

'''Using Open MPI'''

For mixed MPI/OpenMP jobs using OpenMPI, which is the default for many users, the procedure is similar, but details differ.

* Request the number of nodes in the PBS script.
* Set OMP_NUM_THREADS to the number of threads per MPI process.
* In addition to the -np parameter for mpirun, add the argument <tt>--bynode</tt>, so that the mpi processes are not bunched up.

So for example, to start a total of 8 processes on 4 nodes, you could use the following script
<source lang="bash">
#!/bin/bash
# MOAB/Torque submission script for SciNet GPC (hybrid job)
#
#PBS -l nodes=4:ppn=8,walltime=1:00:00
#PBS -N test

# load modules (must match modules used for compilation)
module load intel openmpi

# DIRECTORY TO RUN - $PBS_O_WORKDIR is directory job was submitted from
cd $PBS_O_WORKDIR

# SET THE NUMBER OF THREADS PER PROCESS:
export OMP_NUM_THREADS=4

# EXECUTION COMMAND; -np = nodes*processes_per_nodes; --byhost forces a round robin of nodes.
mpirun -np 8 --bynode ./a.out
</source>

* More questions about running your jobs? See the [[FAQ#Running_your_jobs|FAQs Running your jobs]] and [[FAQ#Errors_in_running_jobs|Errors in running jobs]] sections.

===Submitting an Interactive (Debug) Job===

Your development work flow may require a lot of small test runs. You are allowed to do these on the development nodes, as long as it's very brief (a few minutes), and does not use all cores on the machine. For anything more you will have to use the compute nodes.

It is sometimes convenient to run a job interactively; this can be very handy for debugging purposes. In this case, you type a <tt>qsub</tt> command which submits an interactive job to the queue; when the scheduler selects this job to run, then it starts a shell running on the first node of the job, which connects to your terminal. You can then type any series of commands (for instance, the same commands listed as in the batch submission script above) to run a job interactively.

For example, to start the same sort of job as in the batch submission script above, but interactively, one would type
<pre>
$ qsub -I -l nodes=2:ppn=8,walltime=1:00:00
</pre>

This is exactly the <tt>#PBS -l</tt> line in the batch script above (which requests all 8 processors on each of 2 nodes for one hour), but prepended with a <tt>-I</tt> for `interactive'. When this job begins, your terminal will now show you as being logged in to one of the compute nodes, and one can type in any shell command, run <tt>mpirun</tt>, etc. When you exit the shell, the job will end. Interactive jobs can be used with any of the [[ Moab#GPC | GPC queues ]] however, there is a short
high turnover queue called [[ Moab#debug | debug ]] which can be especially useful when the system is busy.

*More questions about running test jobs? See the [[FAQ#Testing_your_Code|FAQ]].



===QDR vs. DDR Infiniband===

The GPC Infiniband network GPC has two sections, one connecting 3,024 nodes with 5:1 blocking/oversubscribed QDR and the second connecting 840 nodes 1:1 non-blocking DDR Infiniband. By default a user's job will go to whichever network section best accommodates it, typically smaller jobs to the QDR and larger jobs to the DDR. However a user can override this by simply adding the flags "ddr" or "qdr" to the job resource request.

For example, to request two nodes anywhere on the GPC (QDR or DDR), use
<pre>
#PBS -l nodes=2:ppn=8,walltime=1:00:00
</pre>
in your job submission script.

For two nodes using DDR, use
<pre>
#PBS -l nodes=2:ddr:ppn=8,walltime=1:00:00
</pre>

To get two nodes using QDR, instead, you would use
<pre>
#PBS -l nodes=2:qdr:ppn=8,walltime=1:00:00
</pre>

The queueing system also tries its best to keep jobs within the same switch of the QDR thus avoiding the 5:1 blocking. A user can can explicitly request this behaviour if their jobs is less than 30 nodes using

<pre>
#PBS -W x=nodesetisoptional:false
</pre>

===HyperThreading===

Each GPC compute node has 8 Nehalem cores (2 sockets each with a four-core Intel Xeon E5540 @ 2.53GHz). Thus, to make full use of the computing power of a GPC node, you must be running least 8 "tasks" -- MPI processes, or OpenMP threads.

Under most circumstances, running exactly 8 tasks is the most efficient way to use these nodes. However, sometimes software design (eg, having one thread for communication and one for computation) can usefully `oversubscribe' the number of physical cores, and running (say) twice as many tasks as cores can be a useful strategy. If your code is highly memory-bandwidth bound, having one task ready to run while another waits for memory access can make more effective use of the processor.

The Nehalem processors have hardware support for such two-way overloading of processors, through "HyperThreading"; there are an extra set of registers on each core to facilitate rapid switching between two tasks, making it look to the operating system that there are in fact 16 cores per node. Depending on the nature of your code, making use of these virtual extra cores may speed up or slow down your computation; you should run small test cases before running production jobs in this manner. In most cases, the speed difference will be under 10%. Some of our users have obtained an 8% speedup by running gromacs with 16 tasks instead of 8 on a single node (mpirun -np 16 ./gromacs/mdrun -npme 4 is 108% the speed of mpirun -np 8 ./gromacs/mdrun with -npme 2 or -1).

====HyperThreading with OpenMP====

To use hyperthreading with an OpenMP job, one just runs twice as many threads as one would have previously; eg, if you were running 8 threads before (<tt>export OMP_NUM_THREADS=8</tt>) you would run with 16 (<tt>export OMP_NUM_THREADS=16</tt>). Everything else remains the same, including the job submission script; one still uses <tt>ppn=8</tt> in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

====HyperThreading with MPI====

To use hyperthreading with an MPI job, one just runs twice as many MPI processes as one would have previously; eg, if you were running on three nodes using 8 MPI tasks per node and used <tt>mpirun ... -np 24</tt>, you could run instead with <tt>-np 48</tt>. Everything else remains the same, including the job submission script; one still uses <tt>ppn=8</tt> in the submission of the job, as Torque has no way of knowing (or reason for caring) that you will be running on 16 `virtual' cores rather than 8 physical cores.

Note that if you are using OpenMPI (as is the default), there is another consideration; OpenMPI assumes that there is no oversubscription and each task very aggressively makes full use of a core when it is waiting for a message (eg, the waits are "busywaits"). If you find a significant slowdown when running multiple MPI tasks per core with OpenMPI, you may want to try adding the additional option to mpirun: <tt>--mca mpi_yield_when_idle 1</tt>. This will increase the latency of individual messages, but free up the core to do additional work while waiting.

With IntelMPI, the problem should be less pronounced, but you can still improve things by using <tt>mpirun -genv I_MPI_SPIN_COUNT 1 ...</tt>

'''Examples of hyperthreading with MPI'''

Hyperthreading using gromacs: https://support.scinet.utoronto.ca/wiki/index.php/Gromacs#Hyperthreading_with_Gromacs

====HyperThreading with Hybrid MPI/OpenMP codes====

With a hybrid code, one has extra flexibility in how to assign the "extra" cores -- you could run extra MPI tasks or extra OpenMPI threads. As with all hybrid codes, the combination which results in the best performance depends very strongly on the nature of your code, and you should experiment with different combinations. In addition, with hybrid codes processor and memory affinity issues become very important; if you're unsure as to how to tune your application for best performance, please make an appointment with the SciNet technical analysts for more help.

===Memory Configuration===

'''16G'''

There are 3756 nodes which have 16G of memory, and is the primary configuration in the GPC. These nodes will be used by default.
On these nodes, about 2 GB is taken by the operating system. So for mpi runs with 8 processes per node, this leaves about 1.75GB max per mpi process. Do not try to use more than the available memory: the node will crash and your job will either die or hang until the requested walltime has elapsed.

If you need more memory per process or per thread, you can either try to use the limit number of larger memory nodes listed below, or you can run with less mpi processes, or use a different decomposition, such that the job fits on a node.


'''32G'''

There are 84 nodes which have 32G of memory. To request these nodes use:

<pre>
$ qsub -l nodes=2:m32g:ppn=8,walltime=1:00:00
</pre>

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total of 32GB.

'''64G/128G/256G'''

There are 72 16-core Intel Sandybridge nodes which have 64G, 2 with 128G, and 2 with 256G of memory available as part of the contributed [[Sandy]] cluster. These nodes are requested through the <tt>sandy</tt> queue.

<pre>
$ qsub -l nodes=1:m64g:ppn=16,walltime=1:00:00 -q sandy
$ qsub -l nodes=1:m128g:ppn=16,walltime=1:00:00 -q sandy
$ qsub -l nodes=1:m256g:ppn=16,walltime=1:00:00 -q sandy
</pre>

Again, also on these nodes, about 2 GB is taken by the operating system, but this is a relatively small amount compared to the total.

'''128G'''

There are two other stand-alone large memory (128GB) nodes which are primarily to be used for data analysis of runs. They have 16 cores and are intel machines running linux, but they are of a different architecture than the GPC compute nodes, so codes may have to be compiled separately for these machines. This also means that some modules that work on the other GPC nodes, such as octave, will not work in the 128G nodes.

These nodes can be accessed using a specific <tt>largemem</tt> queue.

<pre>
$ qsub -l nodes=2:ppn=16,walltime=1:00:00 -q largemem -I
</pre>

'''Note:''' To estimate your time of access to these nodes, use

<pre>
$ showq -w class=largemem
</pre>

===Ram Disk===

On the GPC nodes, there is a `ram disk' available - up to half of the memory on the node may be used as a temporary file system. This is particularly useful for use in the early stages of migrating destop-computing codes to a High Performance Computing platform such as the GPC. It is much faster than real disk and does not require network traffic; however, each node sees its own ramdisk and cannot see files on that of other nodes. This is a very easy way to cache writes (by writing them to fast ram disk instead of slow `real' disk); and then one would periodically copy the files to files on /scratch or /project so that they are available after the job has completed.

To use the ramdisk, create and read to or write from files in /dev/shm/... just as one would to (eg) $SCRATCH. Only the amount of RAM needed to store the files will be taken up by the temporary file system; thus if you have 8 serial jobs each requiring 1 GB of RAM, and 1GB is taken up by various OS services, you would still have approximately 7GB available to use as ramdisk on a 16GB node. However, if you were to write 8 GB of data to the RAM disk, this would exceed available memory and your job would likely crash.

NOTE: it is very important to delete your files from ram disk at the end of your job. If you do not do this, the next user to use that node will have less RAM available than they might expect, and this might kill their jobs.

More details on how to setup your script to use the ramdisk can be found on the [[User_Ramdisk|Ramdisk wiki page]].

=== Managing jobs on the Queuing system ===

Information on checking available resources, starting, viewing, managing and canceling jobs on [[Moab | Moab/Torque]]. Also check out the [[FAQ#Monitoring_jobs_in_the_queue|FAQ]].

BGQ

2014-07-07T15:01:13Z

Brelier: /* Batch Jobs */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

debugjob session with an executable implicitly calls runjob with 1 mpi task :
<pre>
debugjob -i
**********************************************************
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and
LL14040718574824 for 30 minutes with 64 NODES (1024 cores).
IMPLICIT MODE: running an executable implicitly calls runjob
with 1 mpi task
Exit shell when finished.
**********************************************************
</pre>

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

Oldwiki.scinet.utoronto.ca:System Alerts

2014-07-04T13:03:41Z

Brelier: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|}

Mon Jun 30 15:19:39 EDT: All system down. Some kind of power issue (again).

Sun Jun 29 19:57:29: Compute systems started coming online about 730PM.

Sun Jun 29 18:20:41: filesystems restarted after some issues. Likely at least 8PM before compute systems available

Sun Jun 29 16:39:35 EDT 2014: large voltage spike tripped our main circuit breaker. We have power though it's out at sites within 2k because of lightning strike. Cooling system being restored

Sun Jun 29 15:47:11 EDT 2014: staff enroute to site. Should have update on cause within an hour

Sun Jun 29 15:40:31 EDT 2014: power lost about 3:20P today. All systems down. Investigating.

Note: As a precaution, emails by the Moab/Torque scheduler have been disabled because of a potential security vulnerability since Jan 24th 2014.

Last updated: Fri May 23 12:01:44 EDT 2014
([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2014-07-02T17:21:53Z

Brelier: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|}

Wed Jul 02 13:20:00 EDT : Hadware issue on BlueGene/Q : cannot run full system jobs (2048 nodes)

Mon Jun 30 15:19:39 EDT: All system down. Some kind of power issue (again).

Sun Jun 29 19:57:29: Compute systems started coming online about 730PM.

Sun Jun 29 18:20:41: filesystems restarted after some issues. Likely at least 8PM before compute systems available

Sun Jun 29 16:39:35 EDT 2014: large voltage spike tripped our main circuit breaker. We have power though it's out at sites within 2k because of lightning strike. Cooling system being restored

Sun Jun 29 15:47:11 EDT 2014: staff enroute to site. Should have update on cause within an hour

Sun Jun 29 15:40:31 EDT 2014: power lost about 3:20P today. All systems down. Investigating.

Note: As a precaution, emails by the Moab/Torque scheduler have been disabled because of a potential security vulnerability since Jan 24th 2014.

Last updated: Fri May 23 12:01:44 EDT 2014
([[Previous_messages:|Previous messages]])

Software and Libraries

2014-05-20T13:46:03Z

Brelier: /* GPC Software */

= Software Module System =
All the software listed on this page is accessed using a modules system. This means that much of the software is not
accessible by default but has to be loaded using the module command. The
reason is that
* it allows us to easily keep multiple versions of software for different users on the system;
* it allows users to easily switch between versions.
The module system works similarly on the GPC and the TCS, although different modules are installed on these two systems.

Note that, generally, if you compile a program with a module loaded, you will have to run it with that same module loaded, to make dynamically linked libraries accessible.

{|
!{{Hl2}}|Function
!{{Hl2}}|Command
!{{Hl2}}|Comments
|-
|List available software packages:
|<pre>$ module avail</pre>
|
*If a module is not listed here, it is not supported.
*The flag "(default)" is never part of the name.
|-
|Use particular software:
|<pre> $ module load [module-name] </pre>
|
*If possible, specify only the short name (the part before the "/").
*When ambiguous, this loads the default one.
|-
|List available versions of a specific software package:
|<pre>$ module avail [short-module-name]</pre>
|
|-
|List currently loaded modules:
|<pre>$ module list</pre>
|For reproducability, it is a good idea to put this in your job scripts, so you know exactly what modules(+version) were used.
|-
|Get description of a particular module:
|<pre>$ module help [module-name]</pre>
|
|-
|Remove a module from your shell:
|<pre>$ module unload [module-name]</pre>
|
|-
|Remove all modules:
|<pre>$ module purge</pre>
|
|-
|Replace one loaded module with another:
|<pre>$ module switch [old-module-name] [new-module-name]</pre>
|
|}
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
SCINET_[short-module-name]_BASE
SCINET_[short-module-name]_LIB
SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[short-module-name]_INC}</tt> and <tt>-L${SCINET_[short-module-name]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Errors in loaded modules can arise for a few reasons, for instance:
* A module by that name may not exist.
* Some modules require other modules to have been loaded; it this requirement is not met when you try to load that module, an error message will be printed explaining what module is needed.
* Some modules cannot be loaded together: an error message will be printed explaining which modules conflict.

It is no longer recommended to load modules in the file [[Important_.bashrc_guidelines|.bashrc]] in your home directory, rather, load them explicitly on the command-line and in your job scripts.

== Default and non-default modules ==

When you load a module with its 'short' name, you will get the ''default'' version, which is the most recent (usually), recommended version of that library or piece of software. In general, using the short module name is the way to go. However, you may have code that depends on the intricacies of a non-default version. For that reason, the most common older versions are also available as modules. You can find all available modules using the <tt>module avail</tt> command.

== Naming convention ==

For modules that access applications, the full name of a module is as follows.
<pre>
[short-module-name]/[version-number]
</pre>

To have all modules conform to this convention, a number of modules' name change on Nov 3, 2010:
{|border="1" cellspacing="0" cellpadding="2"
| '''old name'''
| '''new name'''
| '''remarks'''
|-
|autoconf/autoconf-2.64     
|autoconf/2.64
|''short name unchanged''
|-
|cuda/cuda-3.0
|cuda/3.0
|''default's short name unchanged''
|-
|cuda/cuda-3.1
|cuda/3.1
|
|-
|debuggers/ddd-3.3.12
|ddd/3.3.12
|
|-
|debuggers/gdb-7.1
|gdb/7.1
|
|-
|editors/nano/2.2.4
|nano/2.2.4
|
|-
|emacs/emacs-23.1
|emacs/23.1.1
|''short name unchanged''
|-
|gcc/gcc-4.4.0
|gcc/4.4.0
|''short name unchanged''
|-
|graphics/ncview
|ncview/1.93
|
|-
|graphics/graphics
|grace/5.1.22
|
|-
|
|gnuplot/4.2.6
|
|-
|svn/svn165
|svn/1.6.5
|''short name unchanged''
|-
|visualization/paraview
|paraview/3.8
|
|-
|amber10/amber10
|amber/10.0.30
|
|-
|gamess/gamess
|gamess/May2209  
|''default's short name unchanged''
|}

==modulefind - Finding modules by name==

The <tt>module avail</tt> command will only show you modules whose names start with the argument that you give it, and will alsi return modules that you cannot load due to conflicts with already loaded modules.

A little SciNet utility called <tt>modulefind</tt> (one word) can do that. It will list all installed modules which contain the arguments, and will determine whether those modules have
been loaded, could be loaded, cannot because of conflicts with
already loaded modules, or have unresolved dependencies
(i.e. for which other modules need to be loaded first). This is especially useful in cases like the "boost" libraries, whose module names are cxxlibraries/boost/1.47.0-gcc and cxxlibraries/boost/1.47.0-gcc, for the gcc and intel compiler, respectively. <tt>modulefind boost</tt> will find those, whereas <tt>module avail boost</tt> will not.

Note that just 'modulefind' will list all top-level modules.

== Making your own modules ==

How to make your own modules (e.g. for local installations or to access optional perl modules, ...), is possible and described on the [[Installing your own modules]] page.

== Deprecated modules ==

Some older software modules for which newer versions exist, get deprecated, which means they do not get maintained. Since deprecated modules should only be needed in rare exceptional cases, they are not listed by the <tt>module avail</tt> command. However, if you have a piece of legacy code that really depends on a deprecated version of a library (and we urge you to check that it does not work with newer versions!), then you can load a deprecated version by <pre>module load use.deprecated [deprecated-module-name]</pre>

Currently (Oct 5,2010), the following modules are deprecated on the GPC:
<pre>
gcc/gcc-4.3.2 hdf5/184-v16-serial intel/intel-v11.1.046 openmpi/1.3.3-intel-v11.0-ofed
hdf5/183-v16-openmpi hdf5/184-v18-intelmpi intelmpi/impi-3.2.1.009 openmpi/1.3.2-intel-v11.0-ofed.orig
hdf5/183-v18-openmpi hdf5/184-v18-openmpi intelmpi/impi-3.2.2.006 pgplot/5.2.2-gcc.old
hdf5/184-v16-intelmpi hdf5/184-v18-serial intelmpi/impi-4.0.0.013 pgplot/5.2.2-intel.old
hdf5/184-v16-openmpi intel/intel-v11.0.081 intelmpi/impi-4.0.0.025
</pre>

On the TCS, currently (Oct 5,2010) the only deprecated module is:
<pre>
ncl/5.1.1old
</pre>

Before using any of these deprecated modules, make sure that there is not a regular module that satisfies your needs, likely by a ''very similar name''.

== Commercial software ==

Apart from the compilers on our systems and the ddt parallel debugger, we generally do not provide licensed application software, e.g., no Gaussian, IDL, Matlab, etc.
See the [https://support.scinet.utoronto.ca/wiki/index.php/FAQ#How_can_I_run_Matlab_.2F_IDL_.2F_Gaussian_.2F_my_favourite_commercial_software_at_SciNet.3F FAQ].

== Other software and libraries ==

If you want to use a piece of software or a library that is not on the list, you can in principle install it yourself in you <tt>/home</tt> directory.
Note however that building libraries and software from source often uses a lot of files. To avoid running out of disk space, building software is therefore best done from the <tt>/scratch</tt>, from which
you can copy/install only the libraries, header files and binaries to your <tt>/home</tt> directory.

If you suspect that a particular piece of software or a library would be of use to other users of SciNet as well, contact us, and we will consider adding it to the system.

== Software lists ==
=== ARC/GPU Software ===

The CPUs in the GPU nodes of the ARC cluster are of the same kind as those of the GPC, so all modules available on the GPC are available on the GPU nodes with a CentOS 6 image. This means that the different cuda variants that are available as modules, can be loaded on those GPC nodes as well, although they are of little use on that system.

=== GPC Software ===

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Software
!{{Hl2}}| Versions
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|Intel Compiler
|12.1.3*, 12.1.5, 13.1.0
| includes MKL library, which includes BLAS, LAPACK, FFT, ...
| <tt>icpc,icc,ifort</tt>
| <tt>intel</tt>
|-
| GCC Compiler
| 4.4.6, 4.6.1*, 4.7.0, 4.7.2
|
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Cuda
| 3.2, 4.0, 4.1*, 4.2
| NVIDIA's extension to C for GPGPU programming
| <tt>nvcc</tt>
| <tt>cuda</tt>
|-
| PGI Compiler
| 12.5
| supports OpenACC and CUDA Fortran
| <tt>pgcc,pgcpp,pgfortran</tt>
| <tt>pgi</tt>
|-
| IntelMPI
| 4.0.2
| MPICH2 based MPI
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>intelmpi</tt>
|-
| OpenMPI
| 1.4.4*, 1.5.4
|
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>openmpi</tt>
|-
| UPC
| 2.12.2
| Berkley Unified Parallel C Implementation
| <tt>upcc</tt>
| <tt>upc</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Editors'''''
|-
| Nano
| 2.2.4
| Nano's another editor
| <tt>nano</tt>
| <tt>nano</tt>
|-
| Emacs
| 23.1.1
| New version of popular text editor
| <tt>emacs</tt>
| <tt>emacs</tt>
|-
| XEmacs
| 21.4.22
| XEmacs editor
| <tt>xemacs</tt>
| <tt>xemacs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Development tools'''''
|-
| Autoconf
| 2.68
|
| <tt>autoconf, ...</tt>
| <tt>autoconf</tt>
|-
| Automake
| 1.11.2
|
| <tt>aclocal, automake</tt>
| <tt>automake</tt>
|-
| CMake
| 2.8.6
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
| Scons
| 2.0
| Software construction tool
| <tt>scons</tt>
| <tt>scons</tt>
|-
| Git
| 1.7.1
| Revision control system
| <tt>git,gitk</tt>
| <tt>git</tt>
|-
| Intel tools
| 2011
| Intel Code Analysis Tools
| Vtune Amplifier XE, Inspector XE
| <tt>inteltools</tt>
|-
| Mercurial
| 1.8.2
| Version control system (part of the python module!)
| <tt>hg</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug and performance tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool, + MAP MPI Profiler
| <tt>ddt, map</tt>
| <tt>ddt</tt>
|-
| DDD
| 3.3.12
| Data Display Debugger
| <tt>ddd</tt>
| <tt>ddd</tt>
|-
| GDB
| 7.3.1
| GNU debugger (the intel idbc debugger is available by default)
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| MPE2
| 2.4.5
| Multi-Processing Environment with intel + OpenMPI
| <tt>mpecc, mpefc, jumpshot</tt>
| <tt>mpe</tt>
|-
| [[Introduction_To_Performance#OpenSpeedShop_.28profiling.2C_MPI_tracing:_GPC.29 | OpenSpeedShop]]
| 2.0.1
| sampling and MPI tracing
| <tt>openss, ...</tt>
| <tt>openspeedshop</tt>
|-
| [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Scalasca]]
| 1.2
| SCalable performance Analysis of LArge SCale Applications (Compiled with OpenMPI)
| <tt>scalasca</tt>
| <tt>scalasca</tt>
|-
| [http://ipm-hpc.sourceforge.net IPM]
| 0.983
| Integrated Performance Monitors http://ipm-hpc.sourceforge.net/]
| <tt>ipm, ipm_parse, ploticus,...</tt>
| <tt>ipm</tt>
|-
| [[Performance_And_Debugging_Tools:_GPC#Valgrind | Valgrind]]
| 3.6.1
| Memory checking utility
| <tt>valgrind,cachegrind</tt>
| <tt>valgrind</tt>
|-
| Padb
| 3.2
| examine and debug parallel programs
| <tt>padb</tt>
| <tt>padb</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Visualization tools'''''
|-
| Grace
| 5.1.22
| Plotting utility
| <tt>xmgrace</tt>
| <tt>grace</tt>
|-
| Gnuplot
| 4.2.6
| Plotting utility Requires 'extras' module if used on compute nodes.
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| [[ Using_Paraview | ParaView ]]
| 3.12.0
| Scientific visualization, server only
| <tt>pvserver,pvbatch,pvpython</tt>
| <tt>paraview</tt>
|-
| VMD
| 1.9
| Visualization and analysis utility
| <tt>vmd</tt>
| <tt>vmd</tt>
|-
| NCL/NCARG
| 6.0.0
| NCARG graphics and ncl utilities
| <tt>ncl</tt>
| <tt>ncl</tt>
|-
| ROOT
| 5.30.00
| ROOT Analysis Framework from CERN
| <tt>root</tt>
| <tt>ROOT</tt>
|-
| ImageMagick
| 6.6.7
| Image manipulation tools
| <tt>convert,animate,composite,...</tt>
| <tt>ImageMagick</tt>
|-
| PGPLOT
| 5.2.2
| Graphics subroutine library
| <tt>libcpgplot,libpgplot,libtkpgplot</tt>
| <tt>pgplot</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf</tt>
|-
| Ncview
| 2.1.1
| Visualization for NetCDF files
| <tt>ncview</tt>
| <tt>ncview</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, etc.</tt>
| <tt>nco</tt>
|-
| CDO
| 1.5.1
| Climate Data Operators
| <tt>cdo</tt>
| <tt>cdo</tt>
|-
| UDUNITS
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| HDF4
| 4.2.6
| Scientific data storage and retrieval
| <tt>h4fc,hdiff,...,libdf,libsz</tt>
| <tt>hdf4</tt>
|-
| [[Hdf5 | HDF5]]
| 1.8.7-v18*
| Scientific data storage and retrieval, parallel I/O
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5</tt>
|-
| [http://www.arg0.net/encfs EncFS ]
| 1.74
| EncFS provides an encrypted filesystem in user-space, (works ONLY on gpc01..04)
| <tt>encfs</tt>
| <tt>encfs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| [[amber|AMBER 10]]
| Amber 10 + Amber Tools 1.3
| Amber Molecular Dynamics Package
| <tt>sander, sander.MPI</tt>
| <tt>amber</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| [[gamess|GAMESS (US)]]
| August 18, 2011 R1
| General Atomic and Molecular Electronic Structure System
| <tt>rungms</tt>
| <tt>gamess</tt>
|-
| [[gromacs|GROMACS]]
| 4.5.5, 4.5.7, 4.6.2
| GROMACS molecular mechanics, single precision, MPI
| <tt>grompp, mdrun</tt>
| <tt>gromacs</tt>
|-
| [[namd|NAMD]]
| 2.8
| NAMD - Scalable Molecular Dynammics
| <tt>namdmpiexec, namd2</tt>
| <tt>namd</tt>
|-
| [[nwchem|NWChem]]
| 6.0
| NWChem Quantum Chemistry
| <tt>nwchem</tt>
| <tt>nwchem</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 4.3.2, 5.0.3
| Quantum Chemistry
| <tt>pw.x, ...</tt>
| <tt>espresso</tt>
|-
| [http://blast.ncbi.nlm.nih.gov BLAST]
| 2.2.23+
| Basic Local Alignment Search Tool
| <tt>blastn,blastp,blastx,psiblast,tblastn...</tt>
| <tt>blast</tt>
|-
|-
| [http://denovoassembler.sourceforge.net RAY]
| 2.1.0 (small k-mer)
| Parallel de novo genome assemblies
| <tt>Ray</tt>
| <tt>ray</tt>
|-
| [[cpmd|CPMD]]
| 3.13.2
| Carr-Parinello molecular dynamics, MPI
| <tt>cpmd.x</tt>
| <tt>cpmd</tt>
|-
| [[R Statistical Package|R]]
| 2.13.1
| statistical computing
| <tt>R</tt>
| <tt>R</tt>
|-
| Octave
| 3.4.3
| Matlab-like environment
| <tt>octave</tt>
| <tt>octave</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| [http://www.mcs.anl.gov/petsc/petsc-as/ PETSc ]
| 3.1*
| Portable, Extensible Toolkit for Scientific Computation (PETSc)
| <tt>libpetsc, etc.. </tt>
| <tt>petsc</tt>
|-
| BOOST
| 1.47.0
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| [[Armadillo C++ linear algebra library | Armadillo]]
| 3.910.0
| C++ armadillo libraries (implement Matlab-like syntax)
| <tt>armadillo</tt>
| <tt>armadillo</tt>
|-
| [http://www.tacc.utexas.edu/tacc-projects/gotoblas2/ GotoBLAS]
| 1.13
| Optimized BLAS implementation
| <tt>libgoto2</tt>
| <tt>gotoblas</tt>
|-
| GSL
| 1.13*, 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| FFTW
| 3.3
| fast Fourier transform library
''Be careful in combining fftw3 and MKL: you need to link fftw3 first, with'' <tt>-L${SCINET_FFTW_LIB} -lfftw3</tt>, then link MKL
| <tt>libfftw3</tt>
| <tt>fftw</tt>
|-
| LAPACK
|
| Provided by the Intel MKL library
| See http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/
| <tt>intel</tt>
|-
| [http://freshmeat.net/projects/rlog RLog ]
| 1.4
| RLog provides a flexible message logging facility for C++ programs and libraries.
| <tt>librlog</tt>
| <tt>cxxlibraries/rlog</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[GNU Parallel]]
| 2012-10-22
| execute commands in parallel
| <tt>parallel</tt>
| <tt>gnu-parallel</tt>
|-
| [[Python]]
| 2.7.2
| Python programming language
| <tt>python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.3.4
| Python programming language. Modules included : numpy 1.8.0 , scipy 0.13.3 , matlotlib 1.3.1 , ipython 1.2.1 , cython 0.20.1
| <tt>python</tt>
| <tt>python</tt>
|-
| Ruby
| 1.9.1
| Ruby programming language
| <tt>ruby</tt>
| <tt>ruby</tt>
|-
| Java
| 1.6.0
| IBM's Java JRE ad SDK
| <tt>java</tt>, <tt>javac</tt>
| <tt>java</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| Xlibraries
|
| A collection of X graphics libraries and tools
| <tt>xterm</tt>, <tt>xpdf</tt>, ...
| <tt>Xlibraries</tt>
|-
| Extras
|
| A collection of standard linux and home-grown tools
| <tt>bc</tt>, <tt>screen</tt>, <tt>xxdiff</tt>, <tt>modulefind</tt>, <tt>ish</tt>, ...
| <tt>extras</tt>
|}

''* Several versions of this module are installed; listed is the default version.''

=== TCS Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM compilers
|10.1(c/c++) 12.1(fortran)
| See [[TCS Quickstart]]
| <tt>xlc,xlC,xlf,xlc_r,xlC_r,xlf_r</tt>
| ''standard available''
|-
|IBM MPI library
|
| See [[TCS Quickstart]]
| <tt>mpcc,mpCC,mpxlf,mpcc_r,mpCC_r,mpxlf_r</tt>
| ''standard available''
|-
| UPC
| 1.2
| Unified Parallel C
| <tt>xlupc</tt>
| <tt>upc</tt>
|-
|IBM fortran compiler
|13.1, 14.1
| newer version
| <tt>xlf,xlf_r</tt>
| xlf/13.1
|-
|IBM c/c++ compilers
|11.1, 12.1
| new versions
| <tt>xlc,xlC,xlc_r,xlC_r</tt>
| vacpp
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| MPE2
| 1.0.6
| Performance Visualization for Parallel Programs
| <tt>libmpe</tt>
| <tt>mpe</tt>
|-
| Scalasca
| 1.2
| SCalable performance Analysis of LArge SCale Applications
| <tt>scalasca, ...</tt>
| <tt>scalasca</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF4
| 4.2.5
| Scientific data storage and retrieval
| <tt>h4fc, hdiff, ..., libdf, libsz</tt>
| <tt>hdf4</tt>
|-
| HDF5
|
| Scientific data storage and retrieval, parallel I/O Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libhdf5</tt>
| <tt>extras</tt>
|-
| NetCDF + ncview
| 4.0.1*
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf, ncview</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.1.1*
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 3.9.6*
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| FFTW
| 3.2.2
| Fast Fourier transform library Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi,libfftw3</tt>
| <tt>extras</tt>
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| LAPACK+SCALAPACK
| 3.4.2+2.0.2
| Linear algebra package. Note that essl, which comes with the ibm compilers contains a large part of lapack as well.
| <tt>liblapack,libscalapack,libblacs</tt>
| <tt>lapack</tt>
|-
| PetSc
| 3.2
| Portable, Extensible Toolkit for Scientific Computation. With external packages mumps, chaco, hypre, parmetis, prometheus, plapack, superlu, sprng.
| <tt>libpetsc,...</tt>
| <tt>petsc</tt>
|-
| extras
|
| Adds paths to a fuller set of libraries to your user environment compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi, libfftw3, libhdf5, liblapack, ...</tt>
| <tt>extras</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gmake
| 3.82
| GNU's make. Replaces AIX make or gmake 3.80.
| <tt>make</tt>, <tt>gmake</tt>
| <tt>gmake</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| NCL
| 5.1.1
| NCAR Command Language
| <tt>ncl, libncl, ...</tt>
| <tt>ncl</tt>
|-
|}
''* Several versions of this module are installed; listed is the default version.''

=== P7 Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1, 13.1
|See [[P7 Linux Cluster]]
|<tt>xlf,xlf_r,xlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1, 11.1
|See [[P7 Linux Cluster]]
|<tt>xlc,xlC,xlc_r,xlC_r,...</tt>
|vacpp
|-
| Binutils
| 2.23.2
|
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
|IBM MPI library
|5.2.2
|IBM's Parallel Environment
|<tt>mpcc,mpCC,mpfort,mpiexec</tt>
|pe
|-
| GCC Compiler
| 4.6.1 , 4.8.1
| GNU Compiler Collection
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Java
| 7.0
| IBM Java 1.7 implementation
| <tt>javac</tt>
| <tt>jdk</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF5
| 1.8.7
| Scientific data storage and retrieval, parallel I/O
| <tt>libhdf5</tt>
| <tt>hdf5</tt>
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.2.0
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.7.5
| Python programming language. Modules included : numpy-1.8.0 , scipy-0.13.2 , matplotlib-1.3.1 , pyfits-3.2 , h5py-2.2.1
| <tt>python</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gnuplot
| 4.6.1
| command-driven interactive function and data plotting program
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| udunits
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| extras
|
| Adds paths to a fuller set of applications and libraries to your user environment
| <tt>bindlaunch, ...</tt>
| <tt>extras</tt>
|}

=Manuals=
{{:Manuals}}

BGQ

2014-04-07T23:03:55Z

Brelier:

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

The parameter np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

debugjob session with an executable implicitly calls runjob with 1 mpi task :
<pre>
debugjob -i
**********************************************************
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and
LL14040718574824 for 30 minutes with 64 NODES (1024 cores).
IMPLICIT MODE: running an executable implicitly calls runjob
with 1 mpi task
Exit shell when finished.
**********************************************************
</pre>

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-03-28T20:27:32Z

Brelier: /* Software modules installed on the BGQ */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

The parameter np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-03-28T13:29:49Z

Brelier:

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

The parameter np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| first of (20 TB ; 1 million files)
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}
== Python on BlueGene ==
Python 2.7.3 has been installed on BlueGene. To use Numpy and Scipy, the module essl/5.1 has to be loaded.
The full python path has to be provided (otherwise the default version is used).

To use python on BlueGene :
<source lang="bash">
module load python/2.7.3
##Only if you need numpy/scipy :
module load xlf/14.1 essl/5.1
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py
</source>
== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-03-06T17:59:16Z

Brelier: /* Batch Jobs */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

The parameter np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-03-06T17:58:01Z

Brelier: /* Batch Jobs */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

The parameter np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

np : number of MPI processes

ranks-per-node : number of MPI processes per node

OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid code)

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

Software and Libraries

2014-03-04T19:42:48Z

Brelier: /* GPC Software */

= Software Module System =
All the software listed on this page is accessed using a modules system. This means that much of the software is not
accessible by default but has to be loaded using the module command. The
reason is that
* it allows us to easily keep multiple versions of software for different users on the system;
* it allows users to easily switch between versions.
The module system works similarly on the GPC and the TCS, although different modules are installed on these two systems.

Note that, generally, if you compile a program with a module loaded, you will have to run it with that same module loaded, to make dynamically linked libraries accessible.

{|
!{{Hl2}}|Function
!{{Hl2}}|Command
!{{Hl2}}|Comments
|-
|List available software packages:
|<pre>$ module avail</pre>
|
*If a module is not listed here, it is not supported.
*The flag "(default)" is never part of the name.
|-
|Use particular software:
|<pre> $ module load [module-name] </pre>
|
*If possible, specify only the short name (the part before the "/").
*When ambiguous, this loads the default one.
|-
|List available versions of a specific software package:
|<pre>$ module avail [short-module-name]</pre>
|
|-
|List currently loaded modules:
|<pre>$ module list</pre>
|For reproducability, it is a good idea to put this in your job scripts, so you know exactly what modules(+version) were used.
|-
|Get description of a particular module:
|<pre>$ module help [module-name]</pre>
|
|-
|Remove a module from your shell:
|<pre>$ module unload [module-name]</pre>
|
|-
|Remove all modules:
|<pre>$ module purge</pre>
|
|-
|Replace one loaded module with another:
|<pre>$ module switch [old-module-name] [new-module-name]</pre>
|
|}
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
SCINET_[short-module-name]_BASE
SCINET_[short-module-name]_LIB
SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[short-module-name]_INC}</tt> and <tt>-L${SCINET_[short-module-name]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Errors in loaded modules can arise for a few reasons, for instance:
* A module by that name may not exist.
* Some modules require other modules to have been loaded; it this requirement is not met when you try to load that module, an error message will be printed explaining what module is needed.
* Some modules cannot be loaded together: an error message will be printed explaining which modules conflict.

It is no longer recommended to load modules in the file [[Important_.bashrc_guidelines|.bashrc]] in your home directory, rather, load them explicitly on the command-line and in your job scripts.

== Default and non-default modules ==

When you load a module with its 'short' name, you will get the ''default'' version, which is the most recent (usually), recommended version of that library or piece of software. In general, using the short module name is the way to go. However, you may have code that depends on the intricacies of a non-default version. For that reason, the most common older versions are also available as modules. You can find all available modules using the <tt>module avail</tt> command.

== Naming convention ==

For modules that access applications, the full name of a module is as follows.
<pre>
[short-module-name]/[version-number]
</pre>

To have all modules conform to this convention, a number of modules' name change on Nov 3, 2010:
{|border="1" cellspacing="0" cellpadding="2"
| '''old name'''
| '''new name'''
| '''remarks'''
|-
|autoconf/autoconf-2.64     
|autoconf/2.64
|''short name unchanged''
|-
|cuda/cuda-3.0
|cuda/3.0
|''default's short name unchanged''
|-
|cuda/cuda-3.1
|cuda/3.1
|
|-
|debuggers/ddd-3.3.12
|ddd/3.3.12
|
|-
|debuggers/gdb-7.1
|gdb/7.1
|
|-
|editors/nano/2.2.4
|nano/2.2.4
|
|-
|emacs/emacs-23.1
|emacs/23.1.1
|''short name unchanged''
|-
|gcc/gcc-4.4.0
|gcc/4.4.0
|''short name unchanged''
|-
|graphics/ncview
|ncview/1.93
|
|-
|graphics/graphics
|grace/5.1.22
|
|-
|
|gnuplot/4.2.6
|
|-
|svn/svn165
|svn/1.6.5
|''short name unchanged''
|-
|visualization/paraview
|paraview/3.8
|
|-
|amber10/amber10
|amber/10.0.30
|
|-
|gamess/gamess
|gamess/May2209  
|''default's short name unchanged''
|}

==modulefind - Finding modules by name==

The <tt>module avail</tt> command will only show you modules whose names start with the argument that you give it, and will alsi return modules that you cannot load due to conflicts with already loaded modules.

A little SciNet utility called <tt>modulefind</tt> (one word) can do that. It will list all installed modules which contain the arguments, and will determine whether those modules have
been loaded, could be loaded, cannot because of conflicts with
already loaded modules, or have unresolved dependencies
(i.e. for which other modules need to be loaded first). This is especially useful in cases like the "boost" libraries, whose module names are cxxlibraries/boost/1.47.0-gcc and cxxlibraries/boost/1.47.0-gcc, for the gcc and intel compiler, respectively. <tt>modulefind boost</tt> will find those, whereas <tt>module avail boost</tt> will not.

Note that just 'modulefind' will list all top-level modules.

== Making your own modules ==

How to make your own modules (e.g. for local installations or to access optional perl modules, ...), is possible and described on the [[Installing your own modules]] page.

== Deprecated modules ==

Some older software modules for which newer versions exist, get deprecated, which means they do not get maintained. Since deprecated modules should only be needed in rare exceptional cases, they are not listed by the <tt>module avail</tt> command. However, if you have a piece of legacy code that really depends on a deprecated version of a library (and we urge you to check that it does not work with newer versions!), then you can load a deprecated version by <pre>module load use.deprecated [deprecated-module-name]</pre>

Currently (Oct 5,2010), the following modules are deprecated on the GPC:
<pre>
gcc/gcc-4.3.2 hdf5/184-v16-serial intel/intel-v11.1.046 openmpi/1.3.3-intel-v11.0-ofed
hdf5/183-v16-openmpi hdf5/184-v18-intelmpi intelmpi/impi-3.2.1.009 openmpi/1.3.2-intel-v11.0-ofed.orig
hdf5/183-v18-openmpi hdf5/184-v18-openmpi intelmpi/impi-3.2.2.006 pgplot/5.2.2-gcc.old
hdf5/184-v16-intelmpi hdf5/184-v18-serial intelmpi/impi-4.0.0.013 pgplot/5.2.2-intel.old
hdf5/184-v16-openmpi intel/intel-v11.0.081 intelmpi/impi-4.0.0.025
</pre>

On the TCS, currently (Oct 5,2010) the only deprecated module is:
<pre>
ncl/5.1.1old
</pre>

Before using any of these deprecated modules, make sure that there is not a regular module that satisfies your needs, likely by a ''very similar name''.

== Commercial software ==

Apart from the compilers on our systems and the ddt parallel debugger, we generally do not provide licensed application software, e.g., no Gaussian, IDL, Matlab, etc.
See the [https://support.scinet.utoronto.ca/wiki/index.php/FAQ#How_can_I_run_Matlab_.2F_IDL_.2F_Gaussian_.2F_my_favourite_commercial_software_at_SciNet.3F FAQ].

== Other software and libraries ==

If you want to use a piece of software or a library that is not on the list, you can in principle install it yourself in you <tt>/home</tt> directory.
Note however that building libraries and software from source often uses a lot of files. To avoid running out of disk space, building software is therefore best done from the <tt>/scratch</tt>, from which
you can copy/install only the libraries, header files and binaries to your <tt>/home</tt> directory.

If you suspect that a particular piece of software or a library would be of use to other users of SciNet as well, contact us, and we will consider adding it to the system.

== Software lists ==
=== ARC/GPU Software ===

The CPUs in the GPU nodes of the ARC cluster are of the same kind as those of the GPC, so all modules available on the GPC are available on the GPU nodes with a CentOS 6 image. This means that the different cuda variants that are available as modules, can be loaded on those GPC nodes as well, although they are of little use on that system.

=== GPC Software ===

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Software
!{{Hl2}}| Versions
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|Intel Compiler
|12.1.3*, 12.1.5, 13.1.0
| includes MKL library, which includes BLAS, LAPACK, FFT, ...
| <tt>icpc,icc,ifort</tt>
| <tt>intel</tt>
|-
| GCC Compiler
| 4.4.6, 4.6.1*, 4.7.0, 4.7.2
|
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Cuda
| 3.2, 4.0, 4.1*, 4.2
| NVIDIA's extension to C for GPGPU programming
| <tt>nvcc</tt>
| <tt>cuda</tt>
|-
| PGI Compiler
| 12.5
| supports OpenACC and CUDA Fortran
| <tt>pgcc,pgcpp,pgfortran</tt>
| <tt>pgi</tt>
|-
| IntelMPI
| 4.0.2
| MPICH2 based MPI
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>intelmpi</tt>
|-
| OpenMPI
| 1.4.4*, 1.5.4
|
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>openmpi</tt>
|-
| UPC
| 2.12.2
| Berkley Unified Parallel C Implementation
| <tt>upcc</tt>
| <tt>upc</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Editors'''''
|-
| Nano
| 2.2.4
| Nano's another editor
| <tt>nano</tt>
| <tt>nano</tt>
|-
| Emacs
| 23.1.1
| New version of popular text editor
| <tt>emacs</tt>
| <tt>emacs</tt>
|-
| XEmacs
| 21.4.22
| XEmacs editor
| <tt>xemacs</tt>
| <tt>xemacs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Development tools'''''
|-
| Autoconf
| 2.68
|
| <tt>autoconf, ...</tt>
| <tt>autoconf</tt>
|-
| Automake
| 1.11.2
|
| <tt>aclocal, automake</tt>
| <tt>automake</tt>
|-
| CMake
| 2.8.6
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
| Scons
| 2.0
| Software construction tool
| <tt>scons</tt>
| <tt>scons</tt>
|-
| Git
| 1.7.1
| Revision control system
| <tt>git,gitk</tt>
| <tt>git</tt>
|-
| Intel tools
| 2011
| Intel Code Analysis Tools
| Vtune Amplifier XE, Inspector XE
| <tt>inteltools</tt>
|-
| Mercurial
| 1.8.2
| Version control system (part of the python module!)
| <tt>hg</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug and performance tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool, + MAP MPI Profiler
| <tt>ddt, map</tt>
| <tt>ddt</tt>
|-
| DDD
| 3.3.12
| Data Display Debugger
| <tt>ddd</tt>
| <tt>ddd</tt>
|-
| GDB
| 7.3.1
| GNU debugger (the intel idbc debugger is available by default)
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| MPE2
| 2.4.5
| Multi-Processing Environment with intel + OpenMPI
| <tt>mpecc, mpefc, jumpshot</tt>
| <tt>mpe</tt>
|-
| [[Introduction_To_Performance#OpenSpeedShop_.28profiling.2C_MPI_tracing:_GPC.29 | OpenSpeedShop]]
| 2.0.1
| sampling and MPI tracing
| <tt>openss, ...</tt>
| <tt>openspeedshop</tt>
|-
| [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Scalasca]]
| 1.2
| SCalable performance Analysis of LArge SCale Applications (Compiled with OpenMPI)
| <tt>scalasca</tt>
| <tt>scalasca</tt>
|-
| [http://ipm-hpc.sourceforge.net IPM]
| 0.983
| Integrated Performance Monitors http://ipm-hpc.sourceforge.net/]
| <tt>ipm, ipm_parse, ploticus,...</tt>
| <tt>ipm</tt>
|-
| [[Performance_And_Debugging_Tools:_GPC#Valgrind | Valgrind]]
| 3.6.1
| Memory checking utility
| <tt>valgrind,cachegrind</tt>
| <tt>valgrind</tt>
|-
| Padb
| 3.2
| examine and debug parallel programs
| <tt>padb</tt>
| <tt>padb</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Visualization tools'''''
|-
| Grace
| 5.1.22
| Plotting utility
| <tt>xmgrace</tt>
| <tt>grace</tt>
|-
| Gnuplot
| 4.2.6
| Plotting utility Requires 'extras' module if used on compute nodes.
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| [[ Using_Paraview | ParaView ]]
| 3.12.0
| Scientific visualization, server only
| <tt>pvserver,pvbatch,pvpython</tt>
| <tt>paraview</tt>
|-
| VMD
| 1.9
| Visualization and analysis utility
| <tt>vmd</tt>
| <tt>vmd</tt>
|-
| NCL/NCARG
| 6.0.0
| NCARG graphics and ncl utilities
| <tt>ncl</tt>
| <tt>ncl</tt>
|-
| ROOT
| 5.30.00
| ROOT Analysis Framework from CERN
| <tt>root</tt>
| <tt>ROOT</tt>
|-
| ImageMagick
| 6.6.7
| Image manipulation tools
| <tt>convert,animate,composite,...</tt>
| <tt>ImageMagick</tt>
|-
| PGPLOT
| 5.2.2
| Graphics subroutine library
| <tt>libcpgplot,libpgplot,libtkpgplot</tt>
| <tt>pgplot</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf</tt>
|-
| Ncview
| 2.1.1
| Visualization for NetCDF files
| <tt>ncview</tt>
| <tt>ncview</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, etc.</tt>
| <tt>nco</tt>
|-
| CDO
| 1.5.1
| Climate Data Operators
| <tt>cdo</tt>
| <tt>cdo</tt>
|-
| UDUNITS
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| HDF4
| 4.2.6
| Scientific data storage and retrieval
| <tt>h4fc,hdiff,...,libdf,libsz</tt>
| <tt>hdf4</tt>
|-
| [[Hdf5 | HDF5]]
| 1.8.7-v18*
| Scientific data storage and retrieval, parallel I/O
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5</tt>
|-
| [http://www.arg0.net/encfs EncFS ]
| 1.74
| EncFS provides an encrypted filesystem in user-space, (works ONLY on gpc01..04)
| <tt>encfs</tt>
| <tt>encfs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| [[amber|AMBER 10]]
| Amber 10 + Amber Tools 1.3
| Amber Molecular Dynamics Package
| <tt>sander, sander.MPI</tt>
| <tt>amber</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| [[gamess|GAMESS (US)]]
| August 18, 2011 R1
| General Atomic and Molecular Electronic Structure System
| <tt>rungms</tt>
| <tt>gamess</tt>
|-
| [[gromacs|GROMACS]]
| 4.5.5, 4.5.7, 4.6.2
| GROMACS molecular mechanics, single precision, MPI
| <tt>grompp, mdrun</tt>
| <tt>gromacs</tt>
|-
| [[namd|NAMD]]
| 2.8
| NAMD - Scalable Molecular Dynammics
| <tt>namdmpiexec, namd2</tt>
| <tt>namd</tt>
|-
| [[nwchem|NWChem]]
| 6.0
| NWChem Quantum Chemistry
| <tt>nwchem</tt>
| <tt>nwchem</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 4.3.2, 5.0.3
| Quantum Chemistry
| <tt>pw.x, ...</tt>
| <tt>espresso</tt>
|-
| [http://blast.ncbi.nlm.nih.gov BLAST]
| 2.2.23+
| Basic Local Alignment Search Tool
| <tt>blastn,blastp,blastx,psiblast,tblastn...</tt>
| <tt>blast</tt>
|-
|-
| [http://denovoassembler.sourceforge.net RAY]
| 2.1.0 (small k-mer)
| Parallel de novo genome assemblies
| <tt>Ray</tt>
| <tt>ray</tt>
|-
| [[cpmd|CPMD]]
| 3.13.2
| Carr-Parinello molecular dynamics, MPI
| <tt>cpmd.x</tt>
| <tt>cpmd</tt>
|-
| [[R Statistical Package|R]]
| 2.13.1
| statistical computing
| <tt>R</tt>
| <tt>R</tt>
|-
| Octave
| 3.4.3
| Matlab-like environment
| <tt>octave</tt>
| <tt>octave</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| [http://www.mcs.anl.gov/petsc/petsc-as/ PETSc ]
| 3.1*
| Portable, Extensible Toolkit for Scientific Computation (PETSc)
| <tt>libpetsc, etc.. </tt>
| <tt>petsc</tt>
|-
| BOOST
| 1.47.0
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| [[Armadillo C++ linear algebra library | Armadillo]]
| 3.910.0
| C++ armadillo libraries (implement Matlab-like syntax)
| <tt>armadillo</tt>
| <tt>armadillo</tt>
|-
| [http://www.tacc.utexas.edu/tacc-projects/gotoblas2/ GotoBLAS]
| 1.13
| Optimized BLAS implementation
| <tt>libgoto2</tt>
| <tt>gotoblas</tt>
|-
| GSL
| 1.13*, 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| FFTW
| 3.3
| fast Fourier transform library
''Be careful in combining fftw3 and MKL: you need to link fftw3 first, with'' <tt>-L${SCINET_FFTW_LIB} -lfftw3</tt>, then link MKL
| <tt>libfftw3</tt>
| <tt>fftw</tt>
|-
| LAPACK
|
| Provided by the Intel MKL library
| See http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/
| <tt>intel</tt>
|-
| [http://freshmeat.net/projects/rlog RLog ]
| 1.4
| RLog provides a flexible message logging facility for C++ programs and libraries.
| <tt>librlog</tt>
| <tt>cxxlibraries/rlog</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[GNU Parallel]]
| 2012-10-22
| execute commands in parallel
| <tt>parallel</tt>
| <tt>gnu-parallel</tt>
|-
| [[Python]]
| 2.7.2
| Python programming language
| <tt>python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.3.4
| Python programming language. Modules included : numpy 1.8.0 , scipy 0.13.3 , matlotlib 1.3.1 , ipython 1.2.1
| <tt>python</tt>
| <tt>python</tt>
|-
| Ruby
| 1.9.1
| Ruby programming language
| <tt>ruby</tt>
| <tt>ruby</tt>
|-
| Java
| 1.6.0
| IBM's Java JRE ad SDK
| <tt>java</tt>, <tt>javac</tt>
| <tt>java</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| Xlibraries
|
| A collection of X graphics libraries and tools
| <tt>xterm</tt>, <tt>xpdf</tt>, ...
| <tt>Xlibraries</tt>
|-
| Extras
|
| A collection of standard linux and home-grown tools
| <tt>bc</tt>, <tt>screen</tt>, <tt>xxdiff</tt>, <tt>modulefind</tt>, <tt>ish</tt>, ...
| <tt>extras</tt>
|}

''* Several versions of this module are installed; listed is the default version.''

=== TCS Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM compilers
|10.1(c/c++) 12.1(fortran)
| See [[TCS Quickstart]]
| <tt>xlc,xlC,xlf,xlc_r,xlC_r,xlf_r</tt>
| ''standard available''
|-
|IBM MPI library
|
| See [[TCS Quickstart]]
| <tt>mpcc,mpCC,mpxlf,mpcc_r,mpCC_r,mpxlf_r</tt>
| ''standard available''
|-
| UPC
| 1.2
| Unified Parallel C
| <tt>xlupc</tt>
| <tt>upc</tt>
|-
|IBM fortran compiler
|13.1, 14.1
| newer version
| <tt>xlf,xlf_r</tt>
| xlf/13.1
|-
|IBM c/c++ compilers
|11.1, 12.1
| new versions
| <tt>xlc,xlC,xlc_r,xlC_r</tt>
| vacpp
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| MPE2
| 1.0.6
| Performance Visualization for Parallel Programs
| <tt>libmpe</tt>
| <tt>mpe</tt>
|-
| Scalasca
| 1.2
| SCalable performance Analysis of LArge SCale Applications
| <tt>scalasca, ...</tt>
| <tt>scalasca</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF4
| 4.2.5
| Scientific data storage and retrieval
| <tt>h4fc, hdiff, ..., libdf, libsz</tt>
| <tt>hdf4</tt>
|-
| HDF5
|
| Scientific data storage and retrieval, parallel I/O Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libhdf5</tt>
| <tt>extras</tt>
|-
| NetCDF + ncview
| 4.0.1*
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf, ncview</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.1.1*
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 3.9.6*
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| FFTW
| 3.2.2
| Fast Fourier transform library Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi,libfftw3</tt>
| <tt>extras</tt>
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| LAPACK+SCALAPACK
| 3.4.2+2.0.2
| Linear algebra package. Note that essl, which comes with the ibm compilers contains a large part of lapack as well.
| <tt>liblapack,libscalapack,libblacs</tt>
| <tt>lapack</tt>
|-
| PetSc
| 3.2
| Portable, Extensible Toolkit for Scientific Computation. With external packages mumps, chaco, hypre, parmetis, prometheus, plapack, superlu, sprng.
| <tt>libpetsc,...</tt>
| <tt>petsc</tt>
|-
| extras
|
| Adds paths to a fuller set of libraries to your user environment compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi, libfftw3, libhdf5, liblapack, ...</tt>
| <tt>extras</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gmake
| 3.82
| GNU's make. Replaces AIX make or gmake 3.80.
| <tt>make</tt>, <tt>gmake</tt>
| <tt>gmake</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| NCL
| 5.1.1
| NCAR Command Language
| <tt>ncl, libncl, ...</tt>
| <tt>ncl</tt>
|-
|}
''* Several versions of this module are installed; listed is the default version.''

=== P7 Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1, 13.1
|See [[P7 Linux Cluster]]
|<tt>xlf,xlf_r,xlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1, 11.1
|See [[P7 Linux Cluster]]
|<tt>xlc,xlC,xlc_r,xlC_r,...</tt>
|vacpp
|-
| Binutils
| 2.23.2
|
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
|IBM MPI library
|5.2.2
|IBM's Parallel Environment
|<tt>mpcc,mpCC,mpfort,mpiexec</tt>
|pe
|-
| GCC Compiler
| 4.6.1 , 4.8.1
| GNU Compiler Collection
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Java
| 7.0
| IBM Java 1.7 implementation
| <tt>javac</tt>
| <tt>jdk</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF5
| 1.8.7
| Scientific data storage and retrieval, parallel I/O
| <tt>libhdf5</tt>
| <tt>hdf5</tt>
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.2.0
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.7.5
| Python programming language. Modules included : numpy-1.8.0 , scipy-0.13.2 , matplotlib-1.3.1 , pyfits-3.2 , h5py-2.2.1
| <tt>python</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gnuplot
| 4.6.1
| command-driven interactive function and data plotting program
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| udunits
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| extras
|
| Adds paths to a fuller set of applications and libraries to your user environment
| <tt>bindlaunch, ...</tt>
| <tt>extras</tt>
|}

=Manuals=
{{:Manuals}}

Software and Libraries

2014-03-04T19:41:38Z

Brelier: /* GPC Software */

= Software Module System =
All the software listed on this page is accessed using a modules system. This means that much of the software is not
accessible by default but has to be loaded using the module command. The
reason is that
* it allows us to easily keep multiple versions of software for different users on the system;
* it allows users to easily switch between versions.
The module system works similarly on the GPC and the TCS, although different modules are installed on these two systems.

Note that, generally, if you compile a program with a module loaded, you will have to run it with that same module loaded, to make dynamically linked libraries accessible.

{|
!{{Hl2}}|Function
!{{Hl2}}|Command
!{{Hl2}}|Comments
|-
|List available software packages:
|<pre>$ module avail</pre>
|
*If a module is not listed here, it is not supported.
*The flag "(default)" is never part of the name.
|-
|Use particular software:
|<pre> $ module load [module-name] </pre>
|
*If possible, specify only the short name (the part before the "/").
*When ambiguous, this loads the default one.
|-
|List available versions of a specific software package:
|<pre>$ module avail [short-module-name]</pre>
|
|-
|List currently loaded modules:
|<pre>$ module list</pre>
|For reproducability, it is a good idea to put this in your job scripts, so you know exactly what modules(+version) were used.
|-
|Get description of a particular module:
|<pre>$ module help [module-name]</pre>
|
|-
|Remove a module from your shell:
|<pre>$ module unload [module-name]</pre>
|
|-
|Remove all modules:
|<pre>$ module purge</pre>
|
|-
|Replace one loaded module with another:
|<pre>$ module switch [old-module-name] [new-module-name]</pre>
|
|}
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
SCINET_[short-module-name]_BASE
SCINET_[short-module-name]_LIB
SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[short-module-name]_INC}</tt> and <tt>-L${SCINET_[short-module-name]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Errors in loaded modules can arise for a few reasons, for instance:
* A module by that name may not exist.
* Some modules require other modules to have been loaded; it this requirement is not met when you try to load that module, an error message will be printed explaining what module is needed.
* Some modules cannot be loaded together: an error message will be printed explaining which modules conflict.

It is no longer recommended to load modules in the file [[Important_.bashrc_guidelines|.bashrc]] in your home directory, rather, load them explicitly on the command-line and in your job scripts.

== Default and non-default modules ==

When you load a module with its 'short' name, you will get the ''default'' version, which is the most recent (usually), recommended version of that library or piece of software. In general, using the short module name is the way to go. However, you may have code that depends on the intricacies of a non-default version. For that reason, the most common older versions are also available as modules. You can find all available modules using the <tt>module avail</tt> command.

== Naming convention ==

For modules that access applications, the full name of a module is as follows.
<pre>
[short-module-name]/[version-number]
</pre>

To have all modules conform to this convention, a number of modules' name change on Nov 3, 2010:
{|border="1" cellspacing="0" cellpadding="2"
| '''old name'''
| '''new name'''
| '''remarks'''
|-
|autoconf/autoconf-2.64     
|autoconf/2.64
|''short name unchanged''
|-
|cuda/cuda-3.0
|cuda/3.0
|''default's short name unchanged''
|-
|cuda/cuda-3.1
|cuda/3.1
|
|-
|debuggers/ddd-3.3.12
|ddd/3.3.12
|
|-
|debuggers/gdb-7.1
|gdb/7.1
|
|-
|editors/nano/2.2.4
|nano/2.2.4
|
|-
|emacs/emacs-23.1
|emacs/23.1.1
|''short name unchanged''
|-
|gcc/gcc-4.4.0
|gcc/4.4.0
|''short name unchanged''
|-
|graphics/ncview
|ncview/1.93
|
|-
|graphics/graphics
|grace/5.1.22
|
|-
|
|gnuplot/4.2.6
|
|-
|svn/svn165
|svn/1.6.5
|''short name unchanged''
|-
|visualization/paraview
|paraview/3.8
|
|-
|amber10/amber10
|amber/10.0.30
|
|-
|gamess/gamess
|gamess/May2209  
|''default's short name unchanged''
|}

==modulefind - Finding modules by name==

The <tt>module avail</tt> command will only show you modules whose names start with the argument that you give it, and will alsi return modules that you cannot load due to conflicts with already loaded modules.

A little SciNet utility called <tt>modulefind</tt> (one word) can do that. It will list all installed modules which contain the arguments, and will determine whether those modules have
been loaded, could be loaded, cannot because of conflicts with
already loaded modules, or have unresolved dependencies
(i.e. for which other modules need to be loaded first). This is especially useful in cases like the "boost" libraries, whose module names are cxxlibraries/boost/1.47.0-gcc and cxxlibraries/boost/1.47.0-gcc, for the gcc and intel compiler, respectively. <tt>modulefind boost</tt> will find those, whereas <tt>module avail boost</tt> will not.

Note that just 'modulefind' will list all top-level modules.

== Making your own modules ==

How to make your own modules (e.g. for local installations or to access optional perl modules, ...), is possible and described on the [[Installing your own modules]] page.

== Deprecated modules ==

Some older software modules for which newer versions exist, get deprecated, which means they do not get maintained. Since deprecated modules should only be needed in rare exceptional cases, they are not listed by the <tt>module avail</tt> command. However, if you have a piece of legacy code that really depends on a deprecated version of a library (and we urge you to check that it does not work with newer versions!), then you can load a deprecated version by <pre>module load use.deprecated [deprecated-module-name]</pre>

Currently (Oct 5,2010), the following modules are deprecated on the GPC:
<pre>
gcc/gcc-4.3.2 hdf5/184-v16-serial intel/intel-v11.1.046 openmpi/1.3.3-intel-v11.0-ofed
hdf5/183-v16-openmpi hdf5/184-v18-intelmpi intelmpi/impi-3.2.1.009 openmpi/1.3.2-intel-v11.0-ofed.orig
hdf5/183-v18-openmpi hdf5/184-v18-openmpi intelmpi/impi-3.2.2.006 pgplot/5.2.2-gcc.old
hdf5/184-v16-intelmpi hdf5/184-v18-serial intelmpi/impi-4.0.0.013 pgplot/5.2.2-intel.old
hdf5/184-v16-openmpi intel/intel-v11.0.081 intelmpi/impi-4.0.0.025
</pre>

On the TCS, currently (Oct 5,2010) the only deprecated module is:
<pre>
ncl/5.1.1old
</pre>

Before using any of these deprecated modules, make sure that there is not a regular module that satisfies your needs, likely by a ''very similar name''.

== Commercial software ==

Apart from the compilers on our systems and the ddt parallel debugger, we generally do not provide licensed application software, e.g., no Gaussian, IDL, Matlab, etc.
See the [https://support.scinet.utoronto.ca/wiki/index.php/FAQ#How_can_I_run_Matlab_.2F_IDL_.2F_Gaussian_.2F_my_favourite_commercial_software_at_SciNet.3F FAQ].

== Other software and libraries ==

If you want to use a piece of software or a library that is not on the list, you can in principle install it yourself in you <tt>/home</tt> directory.
Note however that building libraries and software from source often uses a lot of files. To avoid running out of disk space, building software is therefore best done from the <tt>/scratch</tt>, from which
you can copy/install only the libraries, header files and binaries to your <tt>/home</tt> directory.

If you suspect that a particular piece of software or a library would be of use to other users of SciNet as well, contact us, and we will consider adding it to the system.

== Software lists ==
=== ARC/GPU Software ===

The CPUs in the GPU nodes of the ARC cluster are of the same kind as those of the GPC, so all modules available on the GPC are available on the GPU nodes with a CentOS 6 image. This means that the different cuda variants that are available as modules, can be loaded on those GPC nodes as well, although they are of little use on that system.

=== GPC Software ===

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Software
!{{Hl2}}| Versions
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|Intel Compiler
|12.1.3*, 12.1.5, 13.1.0
| includes MKL library, which includes BLAS, LAPACK, FFT, ...
| <tt>icpc,icc,ifort</tt>
| <tt>intel</tt>
|-
| GCC Compiler
| 4.4.6, 4.6.1*, 4.7.0, 4.7.2
|
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Cuda
| 3.2, 4.0, 4.1*, 4.2
| NVIDIA's extension to C for GPGPU programming
| <tt>nvcc</tt>
| <tt>cuda</tt>
|-
| PGI Compiler
| 12.5
| supports OpenACC and CUDA Fortran
| <tt>pgcc,pgcpp,pgfortran</tt>
| <tt>pgi</tt>
|-
| IntelMPI
| 4.0.2
| MPICH2 based MPI
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>intelmpi</tt>
|-
| OpenMPI
| 1.4.4*, 1.5.4
|
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>openmpi</tt>
|-
| UPC
| 2.12.2
| Berkley Unified Parallel C Implementation
| <tt>upcc</tt>
| <tt>upc</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Editors'''''
|-
| Nano
| 2.2.4
| Nano's another editor
| <tt>nano</tt>
| <tt>nano</tt>
|-
| Emacs
| 23.1.1
| New version of popular text editor
| <tt>emacs</tt>
| <tt>emacs</tt>
|-
| XEmacs
| 21.4.22
| XEmacs editor
| <tt>xemacs</tt>
| <tt>xemacs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Development tools'''''
|-
| Autoconf
| 2.68
|
| <tt>autoconf, ...</tt>
| <tt>autoconf</tt>
|-
| Automake
| 1.11.2
|
| <tt>aclocal, automake</tt>
| <tt>automake</tt>
|-
| CMake
| 2.8.6
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
| Scons
| 2.0
| Software construction tool
| <tt>scons</tt>
| <tt>scons</tt>
|-
| Git
| 1.7.1
| Revision control system
| <tt>git,gitk</tt>
| <tt>git</tt>
|-
| Intel tools
| 2011
| Intel Code Analysis Tools
| Vtune Amplifier XE, Inspector XE
| <tt>inteltools</tt>
|-
| Mercurial
| 1.8.2
| Version control system (part of the python module!)
| <tt>hg</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug and performance tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool, + MAP MPI Profiler
| <tt>ddt, map</tt>
| <tt>ddt</tt>
|-
| DDD
| 3.3.12
| Data Display Debugger
| <tt>ddd</tt>
| <tt>ddd</tt>
|-
| GDB
| 7.3.1
| GNU debugger (the intel idbc debugger is available by default)
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| MPE2
| 2.4.5
| Multi-Processing Environment with intel + OpenMPI
| <tt>mpecc, mpefc, jumpshot</tt>
| <tt>mpe</tt>
|-
| [[Introduction_To_Performance#OpenSpeedShop_.28profiling.2C_MPI_tracing:_GPC.29 | OpenSpeedShop]]
| 2.0.1
| sampling and MPI tracing
| <tt>openss, ...</tt>
| <tt>openspeedshop</tt>
|-
| [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Scalasca]]
| 1.2
| SCalable performance Analysis of LArge SCale Applications (Compiled with OpenMPI)
| <tt>scalasca</tt>
| <tt>scalasca</tt>
|-
| [http://ipm-hpc.sourceforge.net IPM]
| 0.983
| Integrated Performance Monitors http://ipm-hpc.sourceforge.net/]
| <tt>ipm, ipm_parse, ploticus,...</tt>
| <tt>ipm</tt>
|-
| [[Performance_And_Debugging_Tools:_GPC#Valgrind | Valgrind]]
| 3.6.1
| Memory checking utility
| <tt>valgrind,cachegrind</tt>
| <tt>valgrind</tt>
|-
| Padb
| 3.2
| examine and debug parallel programs
| <tt>padb</tt>
| <tt>padb</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Visualization tools'''''
|-
| Grace
| 5.1.22
| Plotting utility
| <tt>xmgrace</tt>
| <tt>grace</tt>
|-
| Gnuplot
| 4.2.6
| Plotting utility Requires 'extras' module if used on compute nodes.
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| [[ Using_Paraview | ParaView ]]
| 3.12.0
| Scientific visualization, server only
| <tt>pvserver,pvbatch,pvpython</tt>
| <tt>paraview</tt>
|-
| VMD
| 1.9
| Visualization and analysis utility
| <tt>vmd</tt>
| <tt>vmd</tt>
|-
| NCL/NCARG
| 6.0.0
| NCARG graphics and ncl utilities
| <tt>ncl</tt>
| <tt>ncl</tt>
|-
| ROOT
| 5.30.00
| ROOT Analysis Framework from CERN
| <tt>root</tt>
| <tt>ROOT</tt>
|-
| ImageMagick
| 6.6.7
| Image manipulation tools
| <tt>convert,animate,composite,...</tt>
| <tt>ImageMagick</tt>
|-
| PGPLOT
| 5.2.2
| Graphics subroutine library
| <tt>libcpgplot,libpgplot,libtkpgplot</tt>
| <tt>pgplot</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf</tt>
|-
| Ncview
| 2.1.1
| Visualization for NetCDF files
| <tt>ncview</tt>
| <tt>ncview</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, etc.</tt>
| <tt>nco</tt>
|-
| CDO
| 1.5.1
| Climate Data Operators
| <tt>cdo</tt>
| <tt>cdo</tt>
|-
| UDUNITS
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| HDF4
| 4.2.6
| Scientific data storage and retrieval
| <tt>h4fc,hdiff,...,libdf,libsz</tt>
| <tt>hdf4</tt>
|-
| [[Hdf5 | HDF5]]
| 1.8.7-v18*
| Scientific data storage and retrieval, parallel I/O
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5</tt>
|-
| [http://www.arg0.net/encfs EncFS ]
| 1.74
| EncFS provides an encrypted filesystem in user-space, (works ONLY on gpc01..04)
| <tt>encfs</tt>
| <tt>encfs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| [[amber|AMBER 10]]
| Amber 10 + Amber Tools 1.3
| Amber Molecular Dynamics Package
| <tt>sander, sander.MPI</tt>
| <tt>amber</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| [[gamess|GAMESS (US)]]
| August 18, 2011 R1
| General Atomic and Molecular Electronic Structure System
| <tt>rungms</tt>
| <tt>gamess</tt>
|-
| [[gromacs|GROMACS]]
| 4.5.5, 4.5.7, 4.6.2
| GROMACS molecular mechanics, single precision, MPI
| <tt>grompp, mdrun</tt>
| <tt>gromacs</tt>
|-
| [[namd|NAMD]]
| 2.8
| NAMD - Scalable Molecular Dynammics
| <tt>namdmpiexec, namd2</tt>
| <tt>namd</tt>
|-
| [[nwchem|NWChem]]
| 6.0
| NWChem Quantum Chemistry
| <tt>nwchem</tt>
| <tt>nwchem</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 4.3.2, 5.0.3
| Quantum Chemistry
| <tt>pw.x, ...</tt>
| <tt>espresso</tt>
|-
| [http://blast.ncbi.nlm.nih.gov BLAST]
| 2.2.23+
| Basic Local Alignment Search Tool
| <tt>blastn,blastp,blastx,psiblast,tblastn...</tt>
| <tt>blast</tt>
|-
|-
| [http://denovoassembler.sourceforge.net RAY]
| 2.1.0 (small k-mer)
| Parallel de novo genome assemblies
| <tt>Ray</tt>
| <tt>ray</tt>
|-
| [[cpmd|CPMD]]
| 3.13.2
| Carr-Parinello molecular dynamics, MPI
| <tt>cpmd.x</tt>
| <tt>cpmd</tt>
|-
| [[R Statistical Package|R]]
| 2.13.1
| statistical computing
| <tt>R</tt>
| <tt>R</tt>
|-
| Octave
| 3.4.3
| Matlab-like environment
| <tt>octave</tt>
| <tt>octave</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| [http://www.mcs.anl.gov/petsc/petsc-as/ PETSc ]
| 3.1*
| Portable, Extensible Toolkit for Scientific Computation (PETSc)
| <tt>libpetsc, etc.. </tt>
| <tt>petsc</tt>
|-
| BOOST
| 1.47.0
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| [[Armadillo C++ linear algebra library | Armadillo]]
| 3.910.0
| C++ armadillo libraries (implement Matlab-like syntax)
| <tt>armadillo</tt>
| <tt>armadillo</tt>
|-
| [http://www.tacc.utexas.edu/tacc-projects/gotoblas2/ GotoBLAS]
| 1.13
| Optimized BLAS implementation
| <tt>libgoto2</tt>
| <tt>gotoblas</tt>
|-
| GSL
| 1.13*, 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| FFTW
| 3.3
| fast Fourier transform library
''Be careful in combining fftw3 and MKL: you need to link fftw3 first, with'' <tt>-L${SCINET_FFTW_LIB} -lfftw3</tt>, then link MKL
| <tt>libfftw3</tt>
| <tt>fftw</tt>
|-
| LAPACK
|
| Provided by the Intel MKL library
| See http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/
| <tt>intel</tt>
|-
| [http://freshmeat.net/projects/rlog RLog ]
| 1.4
| RLog provides a flexible message logging facility for C++ programs and libraries.
| <tt>librlog</tt>
| <tt>cxxlibraries/rlog</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[GNU Parallel]]
| 2012-10-22
| execute commands in parallel
| <tt>parallel</tt>
| <tt>gnu-parallel</tt>
|-
| [[Python]]
| 2.7.2
| Python programming language
| <tt>python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.3.4
| Python programming language
| <tt>python</tt>
| <tt>python</tt>
|-
| Ruby
| 1.9.1
| Ruby programming language
| <tt>ruby</tt>
| <tt>ruby</tt>
|-
| Java
| 1.6.0
| IBM's Java JRE ad SDK
| <tt>java</tt>, <tt>javac</tt>
| <tt>java</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| Xlibraries
|
| A collection of X graphics libraries and tools
| <tt>xterm</tt>, <tt>xpdf</tt>, ...
| <tt>Xlibraries</tt>
|-
| Extras
|
| A collection of standard linux and home-grown tools
| <tt>bc</tt>, <tt>screen</tt>, <tt>xxdiff</tt>, <tt>modulefind</tt>, <tt>ish</tt>, ...
| <tt>extras</tt>
|}

''* Several versions of this module are installed; listed is the default version.''

=== TCS Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM compilers
|10.1(c/c++) 12.1(fortran)
| See [[TCS Quickstart]]
| <tt>xlc,xlC,xlf,xlc_r,xlC_r,xlf_r</tt>
| ''standard available''
|-
|IBM MPI library
|
| See [[TCS Quickstart]]
| <tt>mpcc,mpCC,mpxlf,mpcc_r,mpCC_r,mpxlf_r</tt>
| ''standard available''
|-
| UPC
| 1.2
| Unified Parallel C
| <tt>xlupc</tt>
| <tt>upc</tt>
|-
|IBM fortran compiler
|13.1, 14.1
| newer version
| <tt>xlf,xlf_r</tt>
| xlf/13.1
|-
|IBM c/c++ compilers
|11.1, 12.1
| new versions
| <tt>xlc,xlC,xlc_r,xlC_r</tt>
| vacpp
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| MPE2
| 1.0.6
| Performance Visualization for Parallel Programs
| <tt>libmpe</tt>
| <tt>mpe</tt>
|-
| Scalasca
| 1.2
| SCalable performance Analysis of LArge SCale Applications
| <tt>scalasca, ...</tt>
| <tt>scalasca</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF4
| 4.2.5
| Scientific data storage and retrieval
| <tt>h4fc, hdiff, ..., libdf, libsz</tt>
| <tt>hdf4</tt>
|-
| HDF5
|
| Scientific data storage and retrieval, parallel I/O Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libhdf5</tt>
| <tt>extras</tt>
|-
| NetCDF + ncview
| 4.0.1*
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf, ncview</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.1.1*
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 3.9.6*
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| FFTW
| 3.2.2
| Fast Fourier transform library Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi,libfftw3</tt>
| <tt>extras</tt>
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| LAPACK+SCALAPACK
| 3.4.2+2.0.2
| Linear algebra package. Note that essl, which comes with the ibm compilers contains a large part of lapack as well.
| <tt>liblapack,libscalapack,libblacs</tt>
| <tt>lapack</tt>
|-
| PetSc
| 3.2
| Portable, Extensible Toolkit for Scientific Computation. With external packages mumps, chaco, hypre, parmetis, prometheus, plapack, superlu, sprng.
| <tt>libpetsc,...</tt>
| <tt>petsc</tt>
|-
| extras
|
| Adds paths to a fuller set of libraries to your user environment compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi, libfftw3, libhdf5, liblapack, ...</tt>
| <tt>extras</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gmake
| 3.82
| GNU's make. Replaces AIX make or gmake 3.80.
| <tt>make</tt>, <tt>gmake</tt>
| <tt>gmake</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| NCL
| 5.1.1
| NCAR Command Language
| <tt>ncl, libncl, ...</tt>
| <tt>ncl</tt>
|-
|}
''* Several versions of this module are installed; listed is the default version.''

=== P7 Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1, 13.1
|See [[P7 Linux Cluster]]
|<tt>xlf,xlf_r,xlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1, 11.1
|See [[P7 Linux Cluster]]
|<tt>xlc,xlC,xlc_r,xlC_r,...</tt>
|vacpp
|-
| Binutils
| 2.23.2
|
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
|IBM MPI library
|5.2.2
|IBM's Parallel Environment
|<tt>mpcc,mpCC,mpfort,mpiexec</tt>
|pe
|-
| GCC Compiler
| 4.6.1 , 4.8.1
| GNU Compiler Collection
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Java
| 7.0
| IBM Java 1.7 implementation
| <tt>javac</tt>
| <tt>jdk</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF5
| 1.8.7
| Scientific data storage and retrieval, parallel I/O
| <tt>libhdf5</tt>
| <tt>hdf5</tt>
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.2.0
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.7.5
| Python programming language. Modules included : numpy-1.8.0 , scipy-0.13.2 , matplotlib-1.3.1 , pyfits-3.2 , h5py-2.2.1
| <tt>python</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gnuplot
| 4.6.1
| command-driven interactive function and data plotting program
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| udunits
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| extras
|
| Adds paths to a fuller set of applications and libraries to your user environment
| <tt>bindlaunch, ...</tt>
| <tt>extras</tt>
|}

=Manuals=
{{:Manuals}}

BGQ

2014-02-13T17:15:45Z

Brelier: /* Batch Jobs */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

The parameter np ≤ ranks-per-node * bg_size

ranks-per-node ≤ np

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-24T17:58:28Z

Brelier: /* Batch Jobs */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

The parameter np ≤ ranks-per-node * bg_size

(ranks-per-node * OMP_NUM_THREADS ) ≤ 64

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-24T17:51:46Z

Brelier: /* Batch Jobs */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-24T17:50:58Z

Brelier: /* Batch Jobs */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

The parameter bg_size can only be equal to 64, 128, 256, 512, 1024 and 2048.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-24T17:37:01Z

Brelier: /* BACKFILL scheduling */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-24T17:31:17Z

Brelier: /* BACKFILL scheduling */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command '''<tt>llAvailableResources</tt>''' gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-24T16:29:04Z

Brelier: /* BACKFILL scheduling */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
On the Devel system : only a debugjob can start immediately

On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-24T15:32:53Z

Brelier:

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== BACKFILL scheduling ===
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command llAvailableResources gives for example :
<pre>
The Devel system is full

On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min
</pre>
=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

Software and Libraries

2014-01-22T15:53:26Z

Brelier: /* P7 Software */

= Software Module System =
All the software listed on this page is accessed using a modules system. This means that much of the software is not
accessible by default but has to be loaded using the module command. The
reason is that
* it allows us to easily keep multiple versions of software for different users on the system;
* it allows users to easily switch between versions.
The module system works similarly on the GPC and the TCS, although different modules are installed on these two systems.

Note that, generally, if you compile a program with a module loaded, you will have to run it with that same module loaded, to make dynamically linked libraries accessible.

{|
!{{Hl2}}|Function
!{{Hl2}}|Command
!{{Hl2}}|Comments
|-
|List available software packages:
|<pre>$ module avail</pre>
|
*If a module is not listed here, it is not supported.
*The flag "(default)" is never part of the name.
|-
|Use particular software:
|<pre> $ module load [module-name] </pre>
|
*If possible, specify only the short name (the part before the "/").
*When ambiguous, this loads the default one.
|-
|List available versions of a specific software package:
|<pre>$ module avail [short-module-name]</pre>
|
|-
|List currently loaded modules:
|<pre>$ module list</pre>
|For reproducability, it is a good idea to put this in your job scripts, so you know exactly what modules(+version) were used.
|-
|Get description of a particular module:
|<pre>$ module help [module-name]</pre>
|
|-
|Remove a module from your shell:
|<pre>$ module unload [module-name]</pre>
|
|-
|Remove all modules:
|<pre>$ module purge</pre>
|
|-
|Replace one loaded module with another:
|<pre>$ module switch [old-module-name] [new-module-name]</pre>
|
|}
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
SCINET_[short-module-name]_BASE
SCINET_[short-module-name]_LIB
SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[short-module-name]_INC}</tt> and <tt>-L${SCINET_[short-module-name]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Errors in loaded modules can arise for a few reasons, for instance:
* A module by that name may not exist.
* Some modules require other modules to have been loaded; it this requirement is not met when you try to load that module, an error message will be printed explaining what module is needed.
* Some modules cannot be loaded together: an error message will be printed explaining which modules conflict.

It is no longer recommended to load modules in the file [[Important_.bashrc_guidelines|.bashrc]] in your home directory, rather, load them explicitly on the command-line and in your job scripts.

== Default and non-default modules ==

When you load a module with its 'short' name, you will get the ''default'' version, which is the most recent (usually), recommended version of that library or piece of software. In general, using the short module name is the way to go. However, you may have code that depends on the intricacies of a non-default version. For that reason, the most common older versions are also available as modules. You can find all available modules using the <tt>module avail</tt> command.

== Naming convention ==

For modules that access applications, the full name of a module is as follows.
<pre>
[short-module-name]/[version-number]
</pre>

To have all modules conform to this convention, a number of modules' name change on Nov 3, 2010:
{|border="1" cellspacing="0" cellpadding="2"
| '''old name'''
| '''new name'''
| '''remarks'''
|-
|autoconf/autoconf-2.64     
|autoconf/2.64
|''short name unchanged''
|-
|cuda/cuda-3.0
|cuda/3.0
|''default's short name unchanged''
|-
|cuda/cuda-3.1
|cuda/3.1
|
|-
|debuggers/ddd-3.3.12
|ddd/3.3.12
|
|-
|debuggers/gdb-7.1
|gdb/7.1
|
|-
|editors/nano/2.2.4
|nano/2.2.4
|
|-
|emacs/emacs-23.1
|emacs/23.1.1
|''short name unchanged''
|-
|gcc/gcc-4.4.0
|gcc/4.4.0
|''short name unchanged''
|-
|graphics/ncview
|ncview/1.93
|
|-
|graphics/graphics
|grace/5.1.22
|
|-
|
|gnuplot/4.2.6
|
|-
|svn/svn165
|svn/1.6.5
|''short name unchanged''
|-
|visualization/paraview
|paraview/3.8
|
|-
|amber10/amber10
|amber/10.0.30
|
|-
|gamess/gamess
|gamess/May2209  
|''default's short name unchanged''
|}

==modulefind - Finding modules by name==

The <tt>module avail</tt> command will only show you modules whose names start with the argument that you give it, and will alsi return modules that you cannot load due to conflicts with already loaded modules.

A little SciNet utility called <tt>modulefind</tt> (one word) can do that. It will list all installed modules which contain the arguments, and will determine whether those modules have
been loaded, could be loaded, cannot because of conflicts with
already loaded modules, or have unresolved dependencies
(i.e. for which other modules need to be loaded first). This is especially useful in cases like the "boost" libraries, whose module names are cxxlibraries/boost/1.47.0-gcc and cxxlibraries/boost/1.47.0-gcc, for the gcc and intel compiler, respectively. <tt>modulefind boost</tt> will find those, whereas <tt>module avail boost</tt> will not.

Note that just 'modulefind' will list all top-level modules.

== Making your own modules ==

How to make your own modules (e.g. for local installations or to access optional perl modules, ...), is possible and described on the [[Installing your own modules]] page.

== Deprecated modules ==

Some older software modules for which newer versions exist, get deprecated, which means they do not get maintained. Since deprecated modules should only be needed in rare exceptional cases, they are not listed by the <tt>module avail</tt> command. However, if you have a piece of legacy code that really depends on a deprecated version of a library (and we urge you to check that it does not work with newer versions!), then you can load a deprecated version by <pre>module load use.deprecated [deprecated-module-name]</pre>

Currently (Oct 5,2010), the following modules are deprecated on the GPC:
<pre>
gcc/gcc-4.3.2 hdf5/184-v16-serial intel/intel-v11.1.046 openmpi/1.3.3-intel-v11.0-ofed
hdf5/183-v16-openmpi hdf5/184-v18-intelmpi intelmpi/impi-3.2.1.009 openmpi/1.3.2-intel-v11.0-ofed.orig
hdf5/183-v18-openmpi hdf5/184-v18-openmpi intelmpi/impi-3.2.2.006 pgplot/5.2.2-gcc.old
hdf5/184-v16-intelmpi hdf5/184-v18-serial intelmpi/impi-4.0.0.013 pgplot/5.2.2-intel.old
hdf5/184-v16-openmpi intel/intel-v11.0.081 intelmpi/impi-4.0.0.025
</pre>

On the TCS, currently (Oct 5,2010) the only deprecated module is:
<pre>
ncl/5.1.1old
</pre>

Before using any of these deprecated modules, make sure that there is not a regular module that satisfies your needs, likely by a ''very similar name''.

== Commercial software ==

Apart from the compilers on our systems and the ddt parallel debugger, we generally do not provide licensed application software, e.g., no Gaussian, IDL, Matlab, etc.
See the [https://support.scinet.utoronto.ca/wiki/index.php/FAQ#How_can_I_run_Matlab_.2F_IDL_.2F_Gaussian_.2F_my_favourite_commercial_software_at_SciNet.3F FAQ].

== Other software and libraries ==

If you want to use a piece of software or a library that is not on the list, you can in principle install it yourself in you <tt>/home</tt> directory.
Note however that building libraries and software from source often uses a lot of files. To avoid running out of disk space, building software is therefore best done from the <tt>/scratch</tt>, from which
you can copy/install only the libraries, header files and binaries to your <tt>/home</tt> directory.

If you suspect that a particular piece of software or a library would be of use to other users of SciNet as well, contact us, and we will consider adding it to the system.

== Software lists ==
=== ARC/GPU Software ===

The CPUs in the GPU nodes of the ARC cluster are of the same kind as those of the GPC, so all modules available on the GPC are available on the GPU nodes with a CentOS 6 image. This means that the different cuda variants that are available as modules, can be loaded on those GPC nodes as well, although they are of little use on that system.

=== GPC Software ===

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}}| Software
!{{Hl2}}| Versions
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|Intel Compiler
|12.1.3*, 12.1.5, 13.1.0
| includes MKL library, which includes BLAS, LAPACK, FFT, ...
| <tt>icpc,icc,ifort</tt>
| <tt>intel</tt>
|-
| GCC Compiler
| 4.4.6, 4.6.1*, 4.7.0, 4.7.2
|
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Cuda
| 3.2, 4.0, 4.1*, 4.2
| NVIDIA's extension to C for GPGPU programming
| <tt>nvcc</tt>
| <tt>cuda</tt>
|-
| PGI Compiler
| 12.5
| supports OpenACC and CUDA Fortran
| <tt>pgcc,pgcpp,pgfortran</tt>
| <tt>pgi</tt>
|-
| IntelMPI
| 4.0.2
| MPICH2 based MPI
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>intelmpi</tt>
|-
| OpenMPI
| 1.4.4*, 1.5.4
|
| <tt>mpicc,mpiCC,mpif77,mpif90</tt>
| <tt>openmpi</tt>
|-
| UPC
| 2.12.2
| Berkley Unified Parallel C Implementation
| <tt>upcc</tt>
| <tt>upc</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Editors'''''
|-
| Nano
| 2.2.4
| Nano's another editor
| <tt>nano</tt>
| <tt>nano</tt>
|-
| Emacs
| 23.1.1
| New version of popular text editor
| <tt>emacs</tt>
| <tt>emacs</tt>
|-
| XEmacs
| 21.4.22
| XEmacs editor
| <tt>xemacs</tt>
| <tt>xemacs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Development tools'''''
|-
| Autoconf
| 2.68
|
| <tt>autoconf, ...</tt>
| <tt>autoconf</tt>
|-
| Automake
| 1.11.2
|
| <tt>aclocal, automake</tt>
| <tt>automake</tt>
|-
| CMake
| 2.8.6
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
| Scons
| 2.0
| Software construction tool
| <tt>scons</tt>
| <tt>scons</tt>
|-
| Git
| 1.7.1
| Revision control system
| <tt>git,gitk</tt>
| <tt>git</tt>
|-
| Intel tools
| 2011
| Intel Code Analysis Tools
| Vtune Amplifier XE, Inspector XE
| <tt>inteltools</tt>
|-
| Mercurial
| 1.8.2
| Version control system (part of the python module!)
| <tt>hg</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug and performance tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool, + MAP MPI Profiler
| <tt>ddt, map</tt>
| <tt>ddt</tt>
|-
| DDD
| 3.3.12
| Data Display Debugger
| <tt>ddd</tt>
| <tt>ddd</tt>
|-
| GDB
| 7.3.1
| GNU debugger (the intel idbc debugger is available by default)
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| MPE2
| 2.4.5
| Multi-Processing Environment with intel + OpenMPI
| <tt>mpecc, mpefc, jumpshot</tt>
| <tt>mpe</tt>
|-
| [[Introduction_To_Performance#OpenSpeedShop_.28profiling.2C_MPI_tracing:_GPC.29 | OpenSpeedShop]]
| 2.0.1
| sampling and MPI tracing
| <tt>openss, ...</tt>
| <tt>openspeedshop</tt>
|-
| [[Introduction_To_Performance#Scalasca_.28profiling.2C_tracing:_TCS.2C_GPC.29 | Scalasca]]
| 1.2
| SCalable performance Analysis of LArge SCale Applications (Compiled with OpenMPI)
| <tt>scalasca</tt>
| <tt>scalasca</tt>
|-
| [http://ipm-hpc.sourceforge.net IPM]
| 0.983
| Integrated Performance Monitors http://ipm-hpc.sourceforge.net/]
| <tt>ipm, ipm_parse, ploticus,...</tt>
| <tt>ipm</tt>
|-
| [[Performance_And_Debugging_Tools:_GPC#Valgrind | Valgrind]]
| 3.6.1
| Memory checking utility
| <tt>valgrind,cachegrind</tt>
| <tt>valgrind</tt>
|-
| Padb
| 3.2
| examine and debug parallel programs
| <tt>padb</tt>
| <tt>padb</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Visualization tools'''''
|-
| Grace
| 5.1.22
| Plotting utility
| <tt>xmgrace</tt>
| <tt>grace</tt>
|-
| Gnuplot
| 4.2.6
| Plotting utility Requires 'extras' module if used on compute nodes.
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| [[ Using_Paraview | ParaView ]]
| 3.12.0
| Scientific visualization, server only
| <tt>pvserver,pvbatch,pvpython</tt>
| <tt>paraview</tt>
|-
| VMD
| 1.9
| Visualization and analysis utility
| <tt>vmd</tt>
| <tt>vmd</tt>
|-
| NCL/NCARG
| 6.0.0
| NCARG graphics and ncl utilities
| <tt>ncl</tt>
| <tt>ncl</tt>
|-
| ROOT
| 5.30.00
| ROOT Analysis Framework from CERN
| <tt>root</tt>
| <tt>ROOT</tt>
|-
| ImageMagick
| 6.6.7
| Image manipulation tools
| <tt>convert,animate,composite,...</tt>
| <tt>ImageMagick</tt>
|-
| PGPLOT
| 5.2.2
| Graphics subroutine library
| <tt>libcpgplot,libpgplot,libtkpgplot</tt>
| <tt>pgplot</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf</tt>
|-
| Ncview
| 2.1.1
| Visualization for NetCDF files
| <tt>ncview</tt>
| <tt>ncview</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, etc.</tt>
| <tt>nco</tt>
|-
| CDO
| 1.5.1
| Climate Data Operators
| <tt>cdo</tt>
| <tt>cdo</tt>
|-
| UDUNITS
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| HDF4
| 4.2.6
| Scientific data storage and retrieval
| <tt>h4fc,hdiff,...,libdf,libsz</tt>
| <tt>hdf4</tt>
|-
| [[Hdf5 | HDF5]]
| 1.8.7-v18*
| Scientific data storage and retrieval, parallel I/O
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5</tt>
|-
| [http://www.arg0.net/encfs EncFS ]
| 1.74
| EncFS provides an encrypted filesystem in user-space, (works ONLY on gpc01..04)
| <tt>encfs</tt>
| <tt>encfs</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| [[amber|AMBER 10]]
| Amber 10 + Amber Tools 1.3
| Amber Molecular Dynamics Package
| <tt>sander, sander.MPI</tt>
| <tt>amber</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| [[gamess|GAMESS (US)]]
| August 18, 2011 R1
| General Atomic and Molecular Electronic Structure System
| <tt>rungms</tt>
| <tt>gamess</tt>
|-
| [[gromacs|GROMACS]]
| 4.5.5, 4.5.7, 4.6.2
| GROMACS molecular mechanics, single precision, MPI
| <tt>grompp, mdrun</tt>
| <tt>gromacs</tt>
|-
| [[namd|NAMD]]
| 2.8
| NAMD - Scalable Molecular Dynammics
| <tt>namdmpiexec, namd2</tt>
| <tt>namd</tt>
|-
| [[nwchem|NWChem]]
| 6.0
| NWChem Quantum Chemistry
| <tt>nwchem</tt>
| <tt>nwchem</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 4.3.2, 5.0.3
| Quantum Chemistry
| <tt>pw.x, ...</tt>
| <tt>espresso</tt>
|-
| [http://blast.ncbi.nlm.nih.gov BLAST]
| 2.2.23+
| Basic Local Alignment Search Tool
| <tt>blastn,blastp,blastx,psiblast,tblastn...</tt>
| <tt>blast</tt>
|-
|-
| [http://denovoassembler.sourceforge.net RAY]
| 2.1.0 (small k-mer)
| Parallel de novo genome assemblies
| <tt>Ray</tt>
| <tt>ray</tt>
|-
| [[cpmd|CPMD]]
| 3.13.2
| Carr-Parinello molecular dynamics, MPI
| <tt>cpmd.x</tt>
| <tt>cpmd</tt>
|-
| [[R Statistical Package|R]]
| 2.13.1
| statistical computing
| <tt>R</tt>
| <tt>R</tt>
|-
| Octave
| 3.4.3
| Matlab-like environment
| <tt>octave</tt>
| <tt>octave</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| [http://www.mcs.anl.gov/petsc/petsc-as/ PETSc ]
| 3.1*
| Portable, Extensible Toolkit for Scientific Computation (PETSc)
| <tt>libpetsc, etc.. </tt>
| <tt>petsc</tt>
|-
| BOOST
| 1.47.0
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| [[Armadillo C++ linear algebra library | Armadillo]]
| 3.910.0
| C++ armadillo libraries (implement Matlab-like syntax)
| <tt>armadillo</tt>
| <tt>armadillo</tt>
|-
| [http://www.tacc.utexas.edu/tacc-projects/gotoblas2/ GotoBLAS]
| 1.13
| Optimized BLAS implementation
| <tt>libgoto2</tt>
| <tt>gotoblas</tt>
|-
| GSL
| 1.13*, 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| FFTW
| 3.3
| fast Fourier transform library
''Be careful in combining fftw3 and MKL: you need to link fftw3 first, with'' <tt>-L${SCINET_FFTW_LIB} -lfftw3</tt>, then link MKL
| <tt>libfftw3</tt>
| <tt>fftw</tt>
|-
| LAPACK
|
| Provided by the Intel MKL library
| See http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/
| <tt>intel</tt>
|-
| [http://freshmeat.net/projects/rlog RLog ]
| 1.4
| RLog provides a flexible message logging facility for C++ programs and libraries.
| <tt>librlog</tt>
| <tt>cxxlibraries/rlog</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[GNU Parallel]]
| 2012-10-22
| execute commands in parallel
| <tt>parallel</tt>
| <tt>gnu-parallel</tt>
|-
| [[Python]]
| 2.7.2
| Python programming language
| <tt>python</tt>
| <tt>python</tt>
|-
| Ruby
| 1.9.1
| Ruby programming language
| <tt>ruby</tt>
| <tt>ruby</tt>
|-
| Java
| 1.6.0
| IBM's Java JRE ad SDK
| <tt>java</tt>, <tt>javac</tt>
| <tt>java</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| Xlibraries
|
| A collection of X graphics libraries and tools
| <tt>xterm</tt>, <tt>xpdf</tt>, ...
| <tt>Xlibraries</tt>
|-
| Extras
|
| A collection of standard linux and home-grown tools
| <tt>bc</tt>, <tt>screen</tt>, <tt>xxdiff</tt>, <tt>modulefind</tt>, <tt>ish</tt>, ...
| <tt>extras</tt>
|}

''* Several versions of this module are installed; listed is the default version.''

=== TCS Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM compilers
|10.1(c/c++) 12.1(fortran)
| See [[TCS Quickstart]]
| <tt>xlc,xlC,xlf,xlc_r,xlC_r,xlf_r</tt>
| ''standard available''
|-
|IBM MPI library
|
| See [[TCS Quickstart]]
| <tt>mpcc,mpCC,mpxlf,mpcc_r,mpCC_r,mpxlf_r</tt>
| ''standard available''
|-
| UPC
| 1.2
| Unified Parallel C
| <tt>xlupc</tt>
| <tt>upc</tt>
|-
|IBM fortran compiler
|13.1, 14.1
| newer version
| <tt>xlf,xlf_r</tt>
| xlf/13.1
|-
|IBM c/c++ compilers
|11.1, 12.1
| new versions
| <tt>xlc,xlC,xlc_r,xlC_r</tt>
| vacpp
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| MPE2
| 1.0.6
| Performance Visualization for Parallel Programs
| <tt>libmpe</tt>
| <tt>mpe</tt>
|-
| Scalasca
| 1.2
| SCalable performance Analysis of LArge SCale Applications
| <tt>scalasca, ...</tt>
| <tt>scalasca</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF4
| 4.2.5
| Scientific data storage and retrieval
| <tt>h4fc, hdiff, ..., libdf, libsz</tt>
| <tt>hdf4</tt>
|-
| HDF5
|
| Scientific data storage and retrieval, parallel I/O Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libhdf5</tt>
| <tt>extras</tt>
|-
| NetCDF + ncview
| 4.0.1*
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf, ncview</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.1.1*
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 3.9.6*
| NCO utilities to manipulate netCDF files
| <tt>ncap, ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| FFTW
| 3.2.2
| Fast Fourier transform library Part of the extras module on the tcs: compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi,libfftw3</tt>
| <tt>extras</tt>
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| LAPACK+SCALAPACK
| 3.4.2+2.0.2
| Linear algebra package. Note that essl, which comes with the ibm compilers contains a large part of lapack as well.
| <tt>liblapack,libscalapack,libblacs</tt>
| <tt>lapack</tt>
|-
| PetSc
| 3.2
| Portable, Extensible Toolkit for Scientific Computation. With external packages mumps, chaco, hypre, parmetis, prometheus, plapack, superlu, sprng.
| <tt>libpetsc,...</tt>
| <tt>petsc</tt>
|-
| extras
|
| Adds paths to a fuller set of libraries to your user environment compile with <tt>-I${SCINET_EXTRAS_INC}</tt> link with <tt>-L${SCINET_EXTRAS_LIB}</tt>
| <tt>libfftw, libfftw_mpi, libfftw3, libhdf5, liblapack, ...</tt>
| <tt>extras</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gmake
| 3.82
| GNU's make. Replaces AIX make or gmake 3.80.
| <tt>make</tt>, <tt>gmake</tt>
| <tt>gmake</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| NCL
| 5.1.1
| NCAR Command Language
| <tt>ncl, libncl, ...</tt>
| <tt>ncl</tt>
|-
|}
''* Several versions of this module are installed; listed is the default version.''

=== P7 Software ===
{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1, 13.1
|See [[P7 Linux Cluster]]
|<tt>xlf,xlf_r,xlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1, 11.1
|See [[P7 Linux Cluster]]
|<tt>xlc,xlC,xlc_r,xlC_r,...</tt>
|vacpp
|-
| Binutils
| 2.23.2
|
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
|IBM MPI library
|5.2.2
|IBM's Parallel Environment
|<tt>mpcc,mpCC,mpfort,mpiexec</tt>
|pe
|-
| GCC Compiler
| 4.6.1 , 4.8.1
| GNU Compiler Collection
| <tt>gcc,g++,gfortran</tt>
| <tt>gcc</tt>
|-
| Java
| 7.0
| IBM Java 1.7 implementation
| <tt>javac</tt>
| <tt>jdk</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performancs tools'''''
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.0, 4.1*
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools and libraries'''''
|-
| HDF5
| 1.8.7
| Scientific data storage and retrieval, parallel I/O
| <tt>libhdf5</tt>
| <tt>hdf5</tt>
|-
| NetCDF
| 4.1.3
| Scientific data storage and retrieval
| <tt>ncdump, ncgen, libnetcdf</tt>
| <tt>netcdf</tt>
|-
| parallel netCDF
| 1.2.0
| Scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
| NCO
| 4.0.8
| NCO utilities to manipulate netCDF files
| <tt>ncap2, ncatted, </tt> etc.
| <tt>nco</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| GSL
| 1.13
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.7.5
| Python programming language. Modules included : numpy-1.8.0 , scipy-0.13.2 , matplotlib-1.3.1 , pyfits-3.2 , h5py-2.2.1
| <tt>python</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Other'''''
|-
| gnuplot
| 4.6.1
| command-driven interactive function and data plotting program
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| antlr
| 2.7.7
| ANother Tool for Language Recognition
| <tt>antlr, antlr-config libantlr, antlr.jar, antlr.py</tt>
| <tt>antlr</tt>
|-
| udunits
| 2.1.11
| unit conversion utilities
| <tt>libudunits2</tt>
| <tt>udunits</tt>
|-
| extras
|
| Adds paths to a fuller set of applications and libraries to your user environment
| <tt>bindlaunch, ...</tt>
| <tt>extras</tt>
|}

=Manuals=
{{:Manuals}}

BGQ

2014-01-12T21:19:43Z

Brelier: /* Queue Limits */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-12AM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs. As a consequence, a 512-node job can only be submitted with a wall time less than 8 hours on the development system.

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-09T14:51:03Z

Brelier: /* Software modules installed on the BGQ */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-8PM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs.

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-09T14:36:00Z

Brelier: /* Software modules installed on the BGQ */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-8PM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs.

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| HDF5
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc* hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-09T14:35:29Z

Brelier: /* Software modules installed on the BGQ */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-8PM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs.

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| 1.8.12-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/1812-v18-serial-gcc* hdf5/1812-v18-mpich2-gcc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

BGQ

2014-01-09T14:20:07Z

Brelier: /* Software modules installed on the BGQ */

{{Infobox Computer
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]
|name=Blue Gene/Q (BGQ)
|installed=August 2012
|operatingsystem= RH6.2, CNK (Linux)
|loginnode= bgqdev-fen1,bgq-fen1
|nnodes= 2048(32,768 cores), 512 (8,192 cores)
|rampernode=16 GB
|corespernode=16 (64 threads)
|interconnect=5D Torus (jobs), QDR Infiniband (I/O)
|vendorcompilers= bgxlc, bgxlf
|queuetype=Loadleveler
}}

==System Status==

The current BGQ system status can be found on the wiki's [[Main Page]].

==SOSCIP==

The BGQ is a Southern Ontario Smart Computing
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the
University of Toronto's SciNet HPC facility. The SOSCIP
multi-university/industry consortium is funded by the Ontario Government
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/strategic-initiatives/southern-ontario-smart-computing-innovation-platform].

== Support Email ==

Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.

==Specifications==

BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK. The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet has 2 BGQ systems, a half-rack 8192 core development system, and a two-rack 32,768 core production system.

[[Image:BlueGeneQHardware2.png‎ |center]]

=== 5D Torus Network ===

The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.

{|border="1" cellspacing="0" cellpadding="2"
| '''Node Boards '''
| '''Compute Nodes'''
| '''Cores'''
| '''Torus Dimensions'''
|-
| 1
| 32
| 512
| 2x2x2x2x2
|-
| 2 (adjacent pairs)
| 64
| 1024
| 2x2x4x2x2
|-
| 4 (quadrants)
| 128
| 2048
| 2x2x4x4x2
|-
| 8 (halves)
| 256
| 4096
| 4x2x4x4x2
|-
| 16 (midplane)
| 512
| 8192
| 4x4x4x4x2
|-
| 32 (1 rack)
| 1024
| 16384
| 4x4x4x8x2
|-
| 64 (2 racks)
| 2048
| 32768
| 4x4x8x8x2
|}

== Login/Devel Nodes ==

The development nodes for the BGQ are '''bgqdev-fen1''' for the half-rack development system and '''bgq-fen1''' for the 2-rack production system.
You can login to them from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside to the half-rack development system using '''bgqdev.scinet.utoronto.ca''', e.g.
<pre>
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X
</pre>
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics. 
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].

These development nodes are Power7 machines running Linux which serve as compilation and submission hosts for the BGQ. Programs are cross-compiled for the BGQ on these nodes and then submitted to the queue using loadleveler.

===Modules and Environment Variables===

To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.

A list of the installed software can be seen on the system by typing
<pre>
$ module avail
</pre>

To load a module (for example, the default version of the intel compilers)
<pre>
$ module load vacpp
</pre>
To unload a module
<pre>
$ module unload vacpp
</pre>
To unload all modules
<pre>
$ module purge
</pre>

These commands can go in your .bashrc files to make sure you are using the correct packages.

Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention
<pre>
$SCINET_[short-module-name]_BASE
$SCINET_[short-module-name]_LIB
$SCINET_[short-module-name]_INC
</pre>
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.

So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>.

Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.

If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).

=== Compilers ===

The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively.

Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading
the '''mpich2''' module.
<pre>
module load mpich2
</pre>

It is recommended to use at least the following flags when compiling and linking
<pre>
-O3 -qarch=qp -qtune=qp
</pre>

If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.

== Job Submission ==

As the BGQ architecture is different from the development nodes, the only way to test your program is to submit a job to the BGQ. Jobs are submitted through loadleveler using '''runjob''' which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below.

=== runjob ===

All BGQ jobs are launced using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. When using loadleveler this is set for you, and you do not have to specify the block name. For example, if your loadleveler script requests 64 nodes, each with 16 cores (for a total of 1024 cores), you run a job with 16 processes per node and 1024 total processes with
<pre>
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in
</pre>
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64.

(Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>")

runjob flags are shown with
<pre>
runjob -h
</pre>

a particularly useful one is

<pre>
--verbose #
</pre>

where # is from 1-7 which can be helpful in debugging an application.

=== How to set ranks-per-node ===

There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per node, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.

Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.

=== Queue Limits ===

The maximum wall_clock_limit on the development '''<tt>bgqdev</tt>''' system is 12 hours and 24 hours on the production '''<tt>bgq</tt>'''. Official SOSCIP porject jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.

A 64 node block is reserved on the devel system for development and interactive testing from 8AM-8PM everyday including weekends. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs.

=== Batch Jobs ===

Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.

<pre>
#!/bin/sh
# @ job_name = bgsample
# @ job_type = bluegene
# @ comment = "BGQ Job By Size"
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Launch all BGQ jobs using runjob
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags
</pre>

To submit to the queue use

<pre>
llsubmit myscript.sh
</pre>

=== Monitoring Jobs ===

To see running jobs

<pre>
llq2
</pre>

or

<pre>
llq -b
</pre>

to cancel a job use

<pre>
llcancel JOBID
</pre>

and to look at details of the bluegene resources use

<pre>
llbgstatus -M all
</pre>

'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''

=== Interactive Use / Debugging ===

As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes.
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs,
however an interactive session is typically beneficial when debugging and developing. As such a
script has been written to allow a session in which runjob can be run interactively. The script
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section.

<pre>
[user@bgqdev-fen1]$ debugjob

[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags

[user@bgqdev-fen1]$ exit
</pre>

For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.

Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables
<pre>
$ export BG_PGM_LAUNCHER=yes
$ export RUNJOB_NP=1
</pre>
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.

=== Sub-block jobs ===

BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used.

Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.

Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER.
<source lang="bash">
#!/bin/bash
# @ job_name = bgsubblock
# @ job_type = bluegene
# @ comment = "BGQ Job SUBBLOCK "
# @ error = $(job_name).$(Host).$(jobid).err
# @ output = $(job_name).$(Host).$(jobid).out
# @ bg_size = 64
# @ wall_clock_limit = 30:00
# @ bg_connectivity = Torus
# @ queue

# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}
# with size of subblocks in nodes (ie similiar to bg_size)

# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)
source subblocks 4

# 16 jobs of 4 each
for (( i=0; i < 16 ; i++)); do
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &
done
wait
</source>
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.

Let us know if you run into any issues with this technique, please contact bgq-support for help.

== Filesystem ==

The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems).

{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | file system
! {{Hl2}} | purpose
! {{Hl2}} | user quota
! {{Hl2}} | backed up
! {{Hl2}} | purged
|-
| /home
| development
| 50 GB
| yes
| never
|-
| /scratch
| computation
| 20 TB
| no
| not currently
|}

===Transfering files===
Although the GPFS file system of the BGQ is shared between the bgq development and production system, except for HPSS
the file system is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ.
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do
<pre>
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .
</pre>
or from a login node you could do
<pre>
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:
</pre>
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back.

Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.

===How much Disk Space Do I have left?===

The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.
<pre>
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]
-h|-?: help
-a: list usages of all members on the group
-u <user>: as another user on your group
-de: include delta information
-plot: create plots of disk usages
</pre>
Note that the information on usage and quota is only updated hourly!

===Bridge to HPSS===

BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]

== Software modules installed on the BGQ ==

{| border="1" cellpadding="10" cellspacing="0"
!{{Hl2}} |Software
!{{Hl2}}| Version
!{{Hl2}}| Comments
!{{Hl2}}| Command/Library
!{{Hl2}}| Module Name
|-
|colspan=5 style='background: #E0E0E0'|'''''Compilers'''''
|-
|IBM fortran compiler
|14.1
|These are cross compilers
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt>
|xlf
|-
|IBM c/c++ compilers
|12.1
|These are cross compilers
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt>
|vacpp
|-
|MPICH2 MPI library
|1.4.1
|There are 4 versions (see BGQ Applications Development document).
|<tt>mpicc,mpicxx,mpif77,mpif90</tt>
|mpich2
|-
| GCC Compiler
| 4.4.6, 4.8.1
| GNU Compiler Collection for BGQ (4.8.1 requires binutils/2.23 to be loaded)
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt>
| <tt>bgqgcc</tt>
|-
| Binutils
| 2.21.1, 2.23
| Cross-compilation utilities
| <tt>addr2line, ar, ld, ...</tt>
| <tt>binutils</tt>
|-
| CMake
| 2.8.8
| cross-platform, open-source build system
| <tt>cmake</tt>
| <tt>cmake</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''
|-
| gdb
| 7.2
| GNU Debugger
| <tt>gdb</tt>
| <tt>gdb</tt>
|-
| [http://www.allinea.com/products/ddt/ DDT]
| 4.1
| Allinea's Distributed Debugging Tool
| <tt>ddt</tt>
| <tt>ddt</tt>
|-
| [[HPCTW]]
| 1.0
| BGQ MPI and Hardware Counters
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt>
| <tt>hptibm</tt>
|-
| [[MemP]]
| 1.0.3
| BGQ Memory Stats
| <tt>libmemP.a </tt>
| <tt>memP</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''
|-
| HDF5
| 1.8.9-v18
| Scientific data storage and retrieval
| <tt>h5ls, h5diff, ..., libhdf5</tt>
| <tt>hdf5/189-v18-serial-xlc* hdf5/189-v18-mpich2-xlc</tt>
|-
| NetCDF
| 4.2.1.1
| Scientific data storage and retrieval
| <tt>ncdump,ncgen,libnetcdf</tt>
| <tt>netcdf/4.2.1.1-serial-xlc* netcdf/4.2.1.1-mpich2-xlc</tt>
|-
| Parallel NetCDF
| 1.3.1
| Parallel scientific data storage and retrieval using MPI-IO
| <tt>libpnetcdf.a</tt>
| <tt>parallel-netcdf</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''
|-
| ESSL
| 5.1
| IBM Engineering and Scientific Subroutine Library (manual below)
| <tt>libesslbg,libesslsmpbg</tt>
| <tt>essl</tt>
|-
| FFTW
| 2.1.5, 3.3.2, 3.1.2-esslwrapper
| Fast fourier transform
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt>
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt>
|-
| LAPACK + ScaLAPACK
| 3.4.2 + 2.0.2
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.
| <tt>liblapack, libscalpack</tt>
| lapack
|-
| GSL
| 1.15
| GNU Scientific Library
| <tt>libgsl, libgslcblas</tt>
| <tt>gsl</tt>
|-
| BOOST
| 1.47.0, 1.54
| C++ Boost libraries
| <tt>libboost...</tt>
| <tt>cxxlibraries/boost</tt>
|-
| bzip2+szip+zlib
| 1.0.6,2.1,1.2.3
| compression libraries
| <tt>libbz2,libz,libsz</tt>
| <tt>compression</tt>
|-
| METIS
| 5.0.2
| Serial Graph Partitioning and Fill-reducing Matrix Ordering
| <tt>libmetis</tt>
| <tt>metis</tt>
|-
| ParMETIS
| 4.0.2
| Parallel graph partitioning and fill-reducing matrix ordering
| <tt>libparmetis</tt>
| <tt>parmetis</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''
|-
| [[Python]]
| 2.6.6
| Python programming language
| <tt>/bgsys/tools/Python-2.6/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 2.7.3
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.1, mpi4py-1.3.1, h5py-2.2.1
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt>
| <tt>python</tt>
|-
| [[Python]]
| 3.2.2
| Python programming language
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt>
| <tt>python</tt>
|-
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''
|-
| gnuplot
| 4.6.1
| interactive plotting program to be run on front-end nodes
| <tt>gnuplot</tt>
| <tt>gnuplot</tt>
|-
| LAMMPS
| Nov 2012
| Molecular Dynamics
| <tt>lmp_bgq</tt>
| <tt>lammps</tt>
|-
| NAMD
| 2.9
| Molecular Dynamics
| <tt>namd2</tt>
| <tt>namd</tt>
|-
| [http://www.quantum-espresso.org/index.php Quantum Espresso]
| 5.0.3
| Molecular Structure / Quantum Chemistry
| <tt>qe_pw.x, etc</tt>
| <tt>espresso</tt>
|-
| [[BGQ_OpenFOAM | OpenFOAM]]
| 2.2.0
| Computational Fluid Dynamics
| <tt>icofoam,etc. </tt>
| <tt>openfoam</tt>
|}

== Documentation ==
#BGQ Day: Intro to Using the BGQ [[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]] [[Media:Bgqintro.pdf|Slides ]]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]
#BGQ Day: BGQ Hardware Overview [[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]] [https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides]  /   [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]

<!-- PUT IN TRAC !!!

=== *Manual Block Creation* ===

To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node

<pre>
bg_console
</pre>

There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the
following command:

<pre>
gen_small_block <blockid> <midplane> <cnodes> <nodeboard>
gen_small_block R00-M0-N03-32 R00-M0 32 N03
</pre>

The block then needs to be booted using:

<pre>
allocate R00-M0-N03-32
</pre>

If those resources are already booted into another block, that block must be freed before the new block can be
allocated.
<pre>
free R00-M0-N03
</pre>

There are many other functions in bg_console:

<pre>
help all
</pre>

The BGQ default nomenclature for hardware is as follows:

<pre>
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore
</pre>

So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.

--!>

Hdf5 table

2014-01-07T17:30:17Z

Brelier: /* Storing tables in HDF5 */

=='''Storing tables in HDF5'''==
The HDF5 Table interface condenses the steps needed to create tables in HDF5. The datatype of the dataset that gets created is of type H5T_COMPOUND. The members of the table can have different datatypes.

===Writting a table using Python (PyTables)===
PyTables is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data.
PyTables is built on top of the HDF5 library, using the Python language and the NumPy package.
The following example shows how to store a table of 10 records with 8 members :
{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | name
! {{Hl2}} | ADCcount
! {{Hl2}} | grid_i
! {{Hl2}} | grid_j
! {{Hl2}} | pressure
! {{Hl2}} | energy
! {{Hl2}} | idnumber
! {{Hl2}} | pressure2
|-
|16-character String
|Unsigned short integer
|32-bit integer
|32-bit integer
|float (single-precision)
|double (double-precision)
|Signed 64-bit integer
|2-dim table of float (2*3)
|-
|}

The script has been run on gpc with the following modules :
<pre>
module load gcc/4.8.1 intel/14.0.0 python/2.7.2 hdf5/1811-v18-serial-gcc
</pre>
PyTable 3.0.0 has been compiled in my scratch directory.
<pre>
from tables import *

class Particle(IsDescription):
name = StringCol(16) # 16-character String
ADCcount = UInt16Col() # Unsigned short integer
grid_i = Int32Col() # 32-bit integer
grid_j = Int32Col() # 32-bit integer
pressure = Float32Col() # float (single-precision)
energy = Float64Col() # double (double-precision)
idnumber = Int64Col() # Signed 64-bit integer
pressure2 = Float32Col(shape=(2,3)) # array of floats (single-precision)

h5file = open_file("tutorial1.h5", mode = "w", title = "Test file")
group = h5file.create_group("/", 'detector', 'Detector information')
table = h5file.create_table(group, 'readout', Particle, "Readout example")
particle = table.row
for i in xrange(10):
particle['name'] = 'Particle: %6d' % (i)
particle['ADCcount'] = (i * 256) % (1 << 16)
particle['grid_i'] = i
particle['grid_j'] = 10 - i
particle['pressure'] = float(i*i)
particle['energy'] = float(particle['pressure'] ** 4)
particle['idnumber'] = i * (2 ** 34)
particle['pressure2'] = [
[0.5+float(i),1.5+float(i),2.5+float(i)],
[-1.5+float(i),-2.5+float(i),-3.5+float(i)]]
# Insert a new particle record
particle.append()

h5file.close()
</pre>

===Reading the table with a C++ code with MPI for parallel programming===
The following example shows how to read the table in a MPI process (each MPI process will read one individual record).
The code has been compiled and tested on BlueGene with the following modules :
<pre>
module load vacpp/12.1 xlf/14.1 mpich2/xl hdf5/189-v18-mpich2-xlc
mpixlcxx -I$SCINET_HDF5_INC -L$SCINET_ZLIB_LIB -L$SCINET_SZIP_LIB -L$SCINET_HDF5_LIB Test.cpp -o Test -lhdf5_hl -lhdf5 -lsz -lz
</pre>
Test.cpp '''(the order of the variables (alphabetical) is important. The c++ code has to read the variables in the same order as in the hdf5 file : check with h5dump YourFile.h5 )''' :
<pre>
#include "hdf5.h"
#include "hdf5_hl.h"
#include <stdlib.h>
#include <iostream>
#include <stdint.h>
#include <mpi.h>
#define NFIELDS (hsize_t) 8
#define H5FILE_NAME "tutorial1.h5"

int main(int argc, char *argv[])
{
// DEF OF SIZE OF VARIABLES TO READ
typedef struct Particle
{
unsigned short int ADCcount;
double energy;
int grid_i;
int grid_j;
long idnumber;
char name[16];
float pressure;
float pressure2[2][3];
} Particle;

/* Calculate the size and the offsets of our struct members in memory */
size_t dst_size = sizeof( Particle );

size_t dst_offset[NFIELDS] = {
HOFFSET( Particle, ADCcount ),
HOFFSET( Particle, energy ),
HOFFSET( Particle, grid_i ),
HOFFSET( Particle, grid_j ),
HOFFSET( Particle, idnumber ),
HOFFSET( Particle, name ),
HOFFSET( Particle, pressure ),
HOFFSET( Particle, pressure2),
};

//////////////////////////////////////////////////////////////////////////////////////////////////////////////
//MPI

//HDF5 APIs definitions
hid_t file_id; /* file and dataset identifiers */
hid_t plist_id; /* property list identifier( access template) */
herr_t status;

// MPI variables
int mpi_size, mpi_rank;
MPI_Comm comm = MPI_COMM_WORLD;
MPI_Info info = MPI_INFO_NULL;

//Initialize MPI
MPI_Init(&argc, &argv);
MPI_Comm_size(comm, &mpi_size);
MPI_Comm_rank(comm, &mpi_rank);

// Set up file access property list with parallel I/O access
plist_id = H5Pcreate(H5P_FILE_ACCESS);//creates a new property list as an instance of some property list class
H5Pset_fapl_mpio(plist_id, comm, info);

// Read file collectively.
file_id = H5Fopen(H5FILE_NAME, H5F_ACC_RDONLY, plist_id);//H5F_ACC_RDONLY : read-only mode

Particle dst_buf[1];
size_t dst_sizes[NFIELDS] = {
sizeof( dst_buf[0].ADCcount),
sizeof( dst_buf[0].energy),
sizeof( dst_buf[0].grid_i),
sizeof( dst_buf[0].grid_j),
sizeof( dst_buf[0].idnumber),
sizeof( dst_buf[0].name),
sizeof( dst_buf[0].pressure),
sizeof( dst_buf[0].pressure2)
};

//READ FRACTION OF TABLE : example reading one record per MPI process
hsize_t start=mpi_rank;//read Record number mpi_rank
hsize_t nrecords=1;//read 1 record
status=H5TBread_records(file_id,"/detector/readout",start,nrecords,dst_size,dst_offset,dst_sizes,dst_buf);

std::cout<<"Rank = "<<mpi_rank
<<" ,ADCcount = "<<dst_buf[0].ADCcount
<<" ,idnumber = "<<dst_buf[0].idnumber
<<" ,grid_i = "<<dst_buf[0].grid_i
<<" ,grid_j = "<<dst_buf[0].grid_j
<<" ,pressure = "<<dst_buf[0].pressure
<<" ,name = "<<dst_buf[0].name
<<" ,energy = "<<dst_buf[0].energy
<<std::endl;
for(int j=0;j<2;j++){
std::cout<<"Rank = "<<mpi_rank<<" : "<<dst_buf[0].pressure2[j][0]<<" "<<dst_buf[0].pressure2[j][1]<<" "<<dst_buf[0].pressure2[j][2]<<std::endl;
}

//Close property list.
H5Pclose(plist_id);

// Close the file.
H5Fclose(file_id);

MPI_Finalize();

return 0;
}
</pre>
The job has been launched on BlueGene :
<pre>
runjob --np 10 --ranks-per-node=1 --envs OMP_NUM_THREADS=1 : /PATH_TO_THE_TEST_DIRECTORY/Test
</pre>
The output of the job :
<pre>
Rank = 9 ,ADCcount = 2304 ,idnumber = 154618822656 ,grid_i = 9 ,grid_j = 1 ,pressure = 81 ,name = Particle: 9 ,energy = 4.30467e+07
Rank = 9 : 9.5 10.5 11.5
Rank = 9 : 7.5 6.5 5.5
Rank = 8 ,ADCcount = 2048 ,idnumber = 137438953472 ,grid_i = 8 ,grid_j = 2 ,pressure = 64 ,name = Particle: 8 ,energy = 1.67772e+07
Rank = 8 : 8.5 9.5 10.5
Rank = 8 : 6.5 5.5 4.5
Rank = 2 ,ADCcount = 512 ,idnumber = 34359738368 ,grid_i = 2 ,grid_j = 8 ,pressure = 4 ,name = Particle: 2 ,energy = 256
Rank = 2 : 2.5 3.5 4.5
Rank = 2 : 0.5 -0.5 -1.5
Rank = 7 ,ADCcount = 1792 ,idnumber = 120259084288 ,grid_i = 7 ,grid_j = 3 ,pressure = 49 ,name = Particle: 7 ,energy = 5.7648e+06
Rank = 7 : 7.5 8.5 9.5
Rank = 7 : 5.5 4.5 3.5
Rank = 4 ,ADCcount = 1024 ,idnumber = 68719476736 ,grid_i = 4 ,grid_j = 6 ,pressure = 16 ,name = Particle: 4 ,energy = 65536
Rank = 4 : 4.5 5.5 6.5
Rank = 4 : 2.5 1.5 0.5
Rank = 5 ,ADCcount = 1280 ,idnumber = 85899345920 ,grid_i = 5 ,grid_j = 5 ,pressure = 25 ,name = Particle: 5 ,energy = 390625
Rank = 5 : 5.5 6.5 7.5
Rank = 5 : 3.5 2.5 1.5
Rank = 6 ,ADCcount = 1536 ,idnumber = 103079215104 ,grid_i = 6 ,grid_j = 4 ,pressure = 36 ,name = Particle: 6 ,energy = 1.67962e+06
Rank = 6 : 6.5 7.5 8.5
Rank = 6 : 4.5 3.5 2.5
Rank = 0 ,ADCcount = 0 ,idnumber = 0 ,grid_i = 0 ,grid_j = 10 ,pressure = 0 ,name = Particle: 0 ,energy = 0
Rank = 0 : 0.5 1.5 2.5
Rank = 0 : -1.5 -2.5 -3.5
Rank = 1 ,ADCcount = 256 ,idnumber = 17179869184 ,grid_i = 1 ,grid_j = 9 ,pressure = 1 ,name = Particle: 1 ,energy = 1
Rank = 1 : 1.5 2.5 3.5
Rank = 1 : -0.5 -1.5 -2.5
Rank = 3 ,ADCcount = 768 ,idnumber = 51539607552 ,grid_i = 3 ,grid_j = 7 ,pressure = 9 ,name = Particle: 3 ,energy = 6561
Rank = 3 : 3.5 4.5 5.5
Rank = 3 : 1.5 0.5 -0.5

</pre>

Hdf5 table

2014-01-07T14:35:46Z

Brelier: /* Reading the table with a C++ code with MPI for parallel programming */

=='''Storing tables in HDF5'''==
The HDF5 Table interface condenses the steps needed to create tables in HDF5. The datatype of the dataset that gets created is of type H5T_COMPOUND. The members of the table can have different datatypes.

===Writting a table using Python (PyTables)===
PyTables is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data.
PyTables is built on top of the HDF5 library, using the Python language and the NumPy package.
The following example shows how to store a table of 10 records with 8 members :
{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | name
! {{Hl2}} | ADCcount
! {{Hl2}} | grid_i
! {{Hl2}} | grid_j
! {{Hl2}} | pressure
! {{Hl2}} | energy
! {{Hl2}} | idnumber
! {{Hl2}} | pressure2
|-
|16-character String
|Unsigned short integer
|32-bit integer
|32-bit integer
|float (single-precision)
|double (double-precision)
|Signed 64-bit integer
|2-dim table of float (2*3)
|-
|}

The script has been run on gpc with the following modules :
<pre>
module load gcc/4.8.1 intel/14.0.0 python/2.7.2 hdf5/1811-v18-serial-gcc
</pre>
PyTable 3.0.0 has been compiled in my scratch directory.
<pre>
from tables import *

class Particle(IsDescription):
name = StringCol(16) # 16-character String
ADCcount = UInt16Col() # Unsigned short integer
grid_i = Int32Col() # 32-bit integer
grid_j = Int32Col() # 32-bit integer
pressure = Float32Col() # float (single-precision)
energy = Float64Col() # double (double-precision)
idnumber = Int64Col() # Signed 64-bit integer
pressure2 = Float32Col(shape=(2,3)) # array of floats (single-precision)

h5file = open_file("tutorial1.h5", mode = "w", title = "Test file")
group = h5file.create_group("/", 'detector', 'Detector information')
table = h5file.create_table(group, 'readout', Particle, "Readout example")
particle = table.row
for i in xrange(10):
particle['name'] = 'Particle: %6d' % (i)
particle['ADCcount'] = (i * 256) % (1 << 16)
particle['grid_i'] = i
particle['grid_j'] = 10 - i
particle['pressure'] = float(i*i)
particle['energy'] = float(particle['pressure'] ** 4)
particle['idnumber'] = i * (2 ** 34)
particle['pressure2'] = [
[0.5+float(i),1.5+float(i),2.5+float(i)],
[-1.5+float(i),-2.5+float(i),-3.5+float(i)]]
# Insert a new particle record
particle.append()

h5file.close()
</pre>

===Reading the table with a C++ code with MPI for parallel programming===
The following example shows how to read the table in a MPI process (each MPI process will read one individual record).
The code has been compiled and tested on BlueGene with the following modules :
<pre>
module load vacpp/12.1 xlf/14.1 mpich2/xl hdf5/189-v18-mpich2-xlc
mpixlcxx -I$SCINET_HDF5_INC -L$SCINET_ZLIB_LIB -L$SCINET_SZIP_LIB -L$SCINET_HDF5_LIB Test.cpp -o Test -lhdf5_hl -lhdf5 -lsz -lz
</pre>
Test.cpp '''(the order of the variables (alphabetical) is important. The c++ code has to read the variables in the same order as in the hdf5 file : check with h5dump YourFile.h5 )''' :
<pre>
#include "hdf5.h"
#include "hdf5_hl.h"
#include <stdlib.h>
#include <iostream>
#include <stdint.h>
#include <mpi.h>
#define NFIELDS (hsize_t) 8
#define H5FILE_NAME "tutorial1.h5"

int main(int argc, char *argv[])
{
// DEF OF SIZE OF VARIABLES TO READ
typedef struct Particle
{
unsigned short int ADCcount;
double energy;
int grid_i;
int grid_j;
long idnumber;
char name[16];
float pressure;
float pressure2[2][3];
} Particle;

/* Calculate the size and the offsets of our struct members in memory */
size_t dst_size = sizeof( Particle );

size_t dst_offset[NFIELDS] = {
HOFFSET( Particle, ADCcount ),
HOFFSET( Particle, energy ),
HOFFSET( Particle, grid_i ),
HOFFSET( Particle, grid_j ),
HOFFSET( Particle, idnumber ),
HOFFSET( Particle, name ),
HOFFSET( Particle, pressure ),
HOFFSET( Particle, pressure2),
};

//////////////////////////////////////////////////////////////////////////////////////////////////////////////
//MPI

//HDF5 APIs definitions
hid_t file_id; /* file and dataset identifiers */
hid_t plist_id; /* property list identifier( access template) */
herr_t status;

// MPI variables
int mpi_size, mpi_rank;
MPI_Comm comm = MPI_COMM_WORLD;
MPI_Info info = MPI_INFO_NULL;

//Initialize MPI
MPI_Init(&argc, &argv);
MPI_Comm_size(comm, &mpi_size);
MPI_Comm_rank(comm, &mpi_rank);

// Set up file access property list with parallel I/O access
plist_id = H5Pcreate(H5P_FILE_ACCESS);//creates a new property list as an instance of some property list class
H5Pset_fapl_mpio(plist_id, comm, info);

// Read file collectively.
file_id = H5Fopen(H5FILE_NAME, H5F_ACC_RDONLY, plist_id);//H5F_ACC_RDONLY : read-only mode

Particle dst_buf[1];
size_t dst_sizes[NFIELDS] = {
sizeof( dst_buf[0].ADCcount),
sizeof( dst_buf[0].energy),
sizeof( dst_buf[0].grid_i),
sizeof( dst_buf[0].grid_j),
sizeof( dst_buf[0].idnumber),
sizeof( dst_buf[0].name),
sizeof( dst_buf[0].pressure),
sizeof( dst_buf[0].pressure2)
};

//READ FRACTION OF TABLE : example reading one record per MPI process
hsize_t start=mpi_rank;//read Record number mpi_rank
hsize_t nrecords=1;//read 1 record
status=H5TBread_records(file_id,"/detector/readout",start,nrecords,dst_size,dst_offset,dst_sizes,dst_buf);

std::cout<<"Rank = "<<mpi_rank
<<" ,ADCcount = "<<dst_buf[0].ADCcount
<<" ,idnumber = "<<dst_buf[0].idnumber
<<" ,grid_i = "<<dst_buf[0].grid_i
<<" ,grid_j = "<<dst_buf[0].grid_j
<<" ,pressure = "<<dst_buf[0].pressure
<<" ,name = "<<dst_buf[0].name
<<" ,energy = "<<dst_buf[0].energy
<<std::endl;
for(int j=0;j<2;j++){
std::cout<<"Rank = "<<mpi_rank<<" : "<<dst_buf[0].pressure2[j][0]<<" "<<dst_buf[0].pressure2[j][1]<<" "<<dst_buf[0].pressure2[j][2]<<std::endl;
}

//Close property list.
H5Pclose(plist_id);

// Close the file.
H5Fclose(file_id);

MPI_Finalize();

return 1;
}
</pre>
The job has been launched on BlueGene :
<pre>
runjob --np 10 --ranks-per-node=1 --envs OMP_NUM_THREADS=1 : /PATH_TO_THE_TEST_DIRECTORY/Test
</pre>
The output of the job :
<pre>
Rank = 9 ,ADCcount = 2304 ,idnumber = 154618822656 ,grid_i = 9 ,grid_j = 1 ,pressure = 81 ,name = Particle: 9 ,energy = 4.30467e+07
Rank = 9 : 9.5 10.5 11.5
Rank = 9 : 7.5 6.5 5.5
Rank = 8 ,ADCcount = 2048 ,idnumber = 137438953472 ,grid_i = 8 ,grid_j = 2 ,pressure = 64 ,name = Particle: 8 ,energy = 1.67772e+07
Rank = 8 : 8.5 9.5 10.5
Rank = 8 : 6.5 5.5 4.5
Rank = 2 ,ADCcount = 512 ,idnumber = 34359738368 ,grid_i = 2 ,grid_j = 8 ,pressure = 4 ,name = Particle: 2 ,energy = 256
Rank = 2 : 2.5 3.5 4.5
Rank = 2 : 0.5 -0.5 -1.5
Rank = 7 ,ADCcount = 1792 ,idnumber = 120259084288 ,grid_i = 7 ,grid_j = 3 ,pressure = 49 ,name = Particle: 7 ,energy = 5.7648e+06
Rank = 7 : 7.5 8.5 9.5
Rank = 7 : 5.5 4.5 3.5
Rank = 4 ,ADCcount = 1024 ,idnumber = 68719476736 ,grid_i = 4 ,grid_j = 6 ,pressure = 16 ,name = Particle: 4 ,energy = 65536
Rank = 4 : 4.5 5.5 6.5
Rank = 4 : 2.5 1.5 0.5
Rank = 5 ,ADCcount = 1280 ,idnumber = 85899345920 ,grid_i = 5 ,grid_j = 5 ,pressure = 25 ,name = Particle: 5 ,energy = 390625
Rank = 5 : 5.5 6.5 7.5
Rank = 5 : 3.5 2.5 1.5
Rank = 6 ,ADCcount = 1536 ,idnumber = 103079215104 ,grid_i = 6 ,grid_j = 4 ,pressure = 36 ,name = Particle: 6 ,energy = 1.67962e+06
Rank = 6 : 6.5 7.5 8.5
Rank = 6 : 4.5 3.5 2.5
Rank = 0 ,ADCcount = 0 ,idnumber = 0 ,grid_i = 0 ,grid_j = 10 ,pressure = 0 ,name = Particle: 0 ,energy = 0
Rank = 0 : 0.5 1.5 2.5
Rank = 0 : -1.5 -2.5 -3.5
Rank = 1 ,ADCcount = 256 ,idnumber = 17179869184 ,grid_i = 1 ,grid_j = 9 ,pressure = 1 ,name = Particle: 1 ,energy = 1
Rank = 1 : 1.5 2.5 3.5
Rank = 1 : -0.5 -1.5 -2.5
Rank = 3 ,ADCcount = 768 ,idnumber = 51539607552 ,grid_i = 3 ,grid_j = 7 ,pressure = 9 ,name = Particle: 3 ,energy = 6561
Rank = 3 : 3.5 4.5 5.5
Rank = 3 : 1.5 0.5 -0.5

</pre>

Hdf5 table

2014-01-07T14:32:56Z

Brelier: /* Reading the table with a C++ code with MPI for parallel programming */

=='''Storing tables in HDF5'''==
The HDF5 Table interface condenses the steps needed to create tables in HDF5. The datatype of the dataset that gets created is of type H5T_COMPOUND. The members of the table can have different datatypes.

===Writting a table using Python (PyTables)===
PyTables is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data.
PyTables is built on top of the HDF5 library, using the Python language and the NumPy package.
The following example shows how to store a table of 10 records with 8 members :
{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | name
! {{Hl2}} | ADCcount
! {{Hl2}} | grid_i
! {{Hl2}} | grid_j
! {{Hl2}} | pressure
! {{Hl2}} | energy
! {{Hl2}} | idnumber
! {{Hl2}} | pressure2
|-
|16-character String
|Unsigned short integer
|32-bit integer
|32-bit integer
|float (single-precision)
|double (double-precision)
|Signed 64-bit integer
|2-dim table of float (2*3)
|-
|}

The script has been run on gpc with the following modules :
<pre>
module load gcc/4.8.1 intel/14.0.0 python/2.7.2 hdf5/1811-v18-serial-gcc
</pre>
PyTable 3.0.0 has been compiled in my scratch directory.
<pre>
from tables import *

class Particle(IsDescription):
name = StringCol(16) # 16-character String
ADCcount = UInt16Col() # Unsigned short integer
grid_i = Int32Col() # 32-bit integer
grid_j = Int32Col() # 32-bit integer
pressure = Float32Col() # float (single-precision)
energy = Float64Col() # double (double-precision)
idnumber = Int64Col() # Signed 64-bit integer
pressure2 = Float32Col(shape=(2,3)) # array of floats (single-precision)

h5file = open_file("tutorial1.h5", mode = "w", title = "Test file")
group = h5file.create_group("/", 'detector', 'Detector information')
table = h5file.create_table(group, 'readout', Particle, "Readout example")
particle = table.row
for i in xrange(10):
particle['name'] = 'Particle: %6d' % (i)
particle['ADCcount'] = (i * 256) % (1 << 16)
particle['grid_i'] = i
particle['grid_j'] = 10 - i
particle['pressure'] = float(i*i)
particle['energy'] = float(particle['pressure'] ** 4)
particle['idnumber'] = i * (2 ** 34)
particle['pressure2'] = [
[0.5+float(i),1.5+float(i),2.5+float(i)],
[-1.5+float(i),-2.5+float(i),-3.5+float(i)]]
# Insert a new particle record
particle.append()

h5file.close()
</pre>

===Reading the table with a C++ code with MPI for parallel programming===
The following example shows how to read the table in a MPI process (each MPI process will read one individual record).
The code has been compiled and tested on BlueGene with the following modules :
<pre>
module load vacpp/12.1 xlf/14.1 mpich2/xl hdf5/189-v18-mpich2-xlc
mpixlcxx -I$SCINET_HDF5_INC -L$SCINET_ZLIB_LIB -L$SCINET_SZIP_LIB -L$SCINET_HDF5_LIB Test.cpp -o Test -lhdf5_hl -lhdf5 -lsz -lz
</pre>
Test.cpp (the order of the variables is important. The c++ code has to read the variables in the same order as in the hdf5 file) :
<pre>
#include "hdf5.h"
#include "hdf5_hl.h"
#include <stdlib.h>
#include <iostream>
#include <stdint.h>
#include <mpi.h>
#define NFIELDS (hsize_t) 8
#define H5FILE_NAME "tutorial1.h5"

int main(int argc, char *argv[])
{
// DEF OF SIZE OF VARIABLES TO READ
typedef struct Particle
{
unsigned short int ADCcount;
double energy;
int grid_i;
int grid_j;
long idnumber;
char name[16];
float pressure;
float pressure2[2][3];
} Particle;

/* Calculate the size and the offsets of our struct members in memory */
size_t dst_size = sizeof( Particle );

size_t dst_offset[NFIELDS] = {
HOFFSET( Particle, ADCcount ),
HOFFSET( Particle, energy ),
HOFFSET( Particle, grid_i ),
HOFFSET( Particle, grid_j ),
HOFFSET( Particle, idnumber ),
HOFFSET( Particle, name ),
HOFFSET( Particle, pressure ),
HOFFSET( Particle, pressure2),
};

//////////////////////////////////////////////////////////////////////////////////////////////////////////////
//MPI

//HDF5 APIs definitions
hid_t file_id; /* file and dataset identifiers */
hid_t plist_id; /* property list identifier( access template) */
herr_t status;

// MPI variables
int mpi_size, mpi_rank;
MPI_Comm comm = MPI_COMM_WORLD;
MPI_Info info = MPI_INFO_NULL;

//Initialize MPI
MPI_Init(&argc, &argv);
MPI_Comm_size(comm, &mpi_size);
MPI_Comm_rank(comm, &mpi_rank);

// Set up file access property list with parallel I/O access
plist_id = H5Pcreate(H5P_FILE_ACCESS);//creates a new property list as an instance of some property list class
H5Pset_fapl_mpio(plist_id, comm, info);

// Read file collectively.
file_id = H5Fopen(H5FILE_NAME, H5F_ACC_RDONLY, plist_id);//H5F_ACC_RDONLY : read-only mode

Particle dst_buf[1];
size_t dst_sizes[NFIELDS] = {
sizeof( dst_buf[0].ADCcount),
sizeof( dst_buf[0].energy),
sizeof( dst_buf[0].grid_i),
sizeof( dst_buf[0].grid_j),
sizeof( dst_buf[0].idnumber),
sizeof( dst_buf[0].name),
sizeof( dst_buf[0].pressure),
sizeof( dst_buf[0].pressure2)
};

//READ FRACTION OF TABLE : example reading one record per MPI process
hsize_t start=mpi_rank;//read Record number mpi_rank
hsize_t nrecords=1;//read 1 record
status=H5TBread_records(file_id,"/detector/readout",start,nrecords,dst_size,dst_offset,dst_sizes,dst_buf);

std::cout<<"Rank = "<<mpi_rank
<<" ,ADCcount = "<<dst_buf[0].ADCcount
<<" ,idnumber = "<<dst_buf[0].idnumber
<<" ,grid_i = "<<dst_buf[0].grid_i
<<" ,grid_j = "<<dst_buf[0].grid_j
<<" ,pressure = "<<dst_buf[0].pressure
<<" ,name = "<<dst_buf[0].name
<<" ,energy = "<<dst_buf[0].energy
<<std::endl;
for(int j=0;j<2;j++){
std::cout<<"Rank = "<<mpi_rank<<" : "<<dst_buf[0].pressure2[j][0]<<" "<<dst_buf[0].pressure2[j][1]<<" "<<dst_buf[0].pressure2[j][2]<<std::endl;
}

//Close property list.
H5Pclose(plist_id);

// Close the file.
H5Fclose(file_id);

MPI_Finalize();

return 1;
}
</pre>
The job has been launched on BlueGene :
<pre>
runjob --np 10 --ranks-per-node=1 --envs OMP_NUM_THREADS=1 : /PATH_TO_THE_TEST_DIRECTORY/Test
</pre>
The output of the job :
<pre>
Rank = 9 ,ADCcount = 2304 ,idnumber = 154618822656 ,grid_i = 9 ,grid_j = 1 ,pressure = 81 ,name = Particle: 9 ,energy = 4.30467e+07
Rank = 9 : 9.5 10.5 11.5
Rank = 9 : 7.5 6.5 5.5
Rank = 8 ,ADCcount = 2048 ,idnumber = 137438953472 ,grid_i = 8 ,grid_j = 2 ,pressure = 64 ,name = Particle: 8 ,energy = 1.67772e+07
Rank = 8 : 8.5 9.5 10.5
Rank = 8 : 6.5 5.5 4.5
Rank = 2 ,ADCcount = 512 ,idnumber = 34359738368 ,grid_i = 2 ,grid_j = 8 ,pressure = 4 ,name = Particle: 2 ,energy = 256
Rank = 2 : 2.5 3.5 4.5
Rank = 2 : 0.5 -0.5 -1.5
Rank = 7 ,ADCcount = 1792 ,idnumber = 120259084288 ,grid_i = 7 ,grid_j = 3 ,pressure = 49 ,name = Particle: 7 ,energy = 5.7648e+06
Rank = 7 : 7.5 8.5 9.5
Rank = 7 : 5.5 4.5 3.5
Rank = 4 ,ADCcount = 1024 ,idnumber = 68719476736 ,grid_i = 4 ,grid_j = 6 ,pressure = 16 ,name = Particle: 4 ,energy = 65536
Rank = 4 : 4.5 5.5 6.5
Rank = 4 : 2.5 1.5 0.5
Rank = 5 ,ADCcount = 1280 ,idnumber = 85899345920 ,grid_i = 5 ,grid_j = 5 ,pressure = 25 ,name = Particle: 5 ,energy = 390625
Rank = 5 : 5.5 6.5 7.5
Rank = 5 : 3.5 2.5 1.5
Rank = 6 ,ADCcount = 1536 ,idnumber = 103079215104 ,grid_i = 6 ,grid_j = 4 ,pressure = 36 ,name = Particle: 6 ,energy = 1.67962e+06
Rank = 6 : 6.5 7.5 8.5
Rank = 6 : 4.5 3.5 2.5
Rank = 0 ,ADCcount = 0 ,idnumber = 0 ,grid_i = 0 ,grid_j = 10 ,pressure = 0 ,name = Particle: 0 ,energy = 0
Rank = 0 : 0.5 1.5 2.5
Rank = 0 : -1.5 -2.5 -3.5
Rank = 1 ,ADCcount = 256 ,idnumber = 17179869184 ,grid_i = 1 ,grid_j = 9 ,pressure = 1 ,name = Particle: 1 ,energy = 1
Rank = 1 : 1.5 2.5 3.5
Rank = 1 : -0.5 -1.5 -2.5
Rank = 3 ,ADCcount = 768 ,idnumber = 51539607552 ,grid_i = 3 ,grid_j = 7 ,pressure = 9 ,name = Particle: 3 ,energy = 6561
Rank = 3 : 3.5 4.5 5.5
Rank = 3 : 1.5 0.5 -0.5

</pre>

NetCDF table

2014-01-06T14:38:20Z

Brelier: /* Using Parallel netCDF */

=='''Storing tables in NetCDF4'''==

===Writting a table using Python ===
The following example shows how to store a table of 10 records with 8 members :
{|border="1" cellpadding="10" cellspacing="0"
|-
! {{Hl2}} | name
! {{Hl2}} | ADCcount
! {{Hl2}} | grid_i
! {{Hl2}} | grid_j
! {{Hl2}} | pressure
! {{Hl2}} | energy
! {{Hl2}} | idnumber
! {{Hl2}} | pressure2
|-
|16-character String
|Unsigned short integer
|32-bit integer
|32-bit integer
|float (single-precision)
|double (double-precision)
|Signed 64-bit integer
|2-dim table of float (2*3)
|-
|}

The script has been run on gpc with the following modules :
<pre>
module load gcc/4.8.1 hdf5/187-v18-serial-gcc netcdf/4.1.3_hdf5_serial-gcc intel/14.0.0 python/2.7.2
</pre>
<pre>
from netCDF4 import Dataset
from netCDF4 import chartostring, stringtoarr
import numpy

f = Dataset('particles.nc','w',format='NETCDF4')

size = 10

Particle = numpy.dtype([('name', 'S1', 16), # 16-character String
('ADCcount',numpy.uint16), # Unsigned short integer
('grid_i',numpy.int32), # 32-bit integer
('grid_j',numpy.int32), # 32-bit integer
('pressure',numpy.float32), # float (single-precision)
('energy',numpy.float64), # double (double-precision)
('idnumber',numpy.int64), # Signed 64-bit integer
('pressure2' , numpy.float32 , (2,3) ) # array of floats (single-precision) table 2 lines * 3 columns
])

Particle_t = f.createCompoundType(Particle,'Particle')

f.createDimension('NRecords',None)
v = f.createVariable('Data',Particle_t,'NRecords')
data = numpy.empty(size,Particle)

for i in xrange(10):
data['name'][i] = stringtoarr('Particle: %6d' % (i),16)
data['ADCcount'][i] = (i * 256) % (1 << 16)
data['grid_i'][i] = i
data['grid_j'][i] = 10 - i
data['pressure'][i] = float(i*i)
data['energy'][i] = float(data['pressure'][i] ** 4)
data['idnumber'][i] = i * (2 ** 34)
data['pressure2'][i] = [
[0.5+float(i),1.5+float(i),2.5+float(i)],
[-1.5+float(i),-2.5+float(i),-3.5+float(i)]]
#Fill data in File
v[:] = data

f.close()
</pre>
The NetCDF file can be dumped using : ncdump particles.nc :
<pre>
netcdf particles {
types:
compound Particle {
char name(16) ;
ushort ADCcount ;
int grid_i ;
int grid_j ;
float pressure ;
double energy ;
int64 idnumber ;
float pressure2(2, 3) ;
}; // Particle
dimensions:
NRecords = UNLIMITED ; // (10 currently)
variables:
Particle Data(NRecords) ;
data:

Data =
{{"Particle: 0"}, 0, 0, 10, 0, 0, 0, {0.5, 1.5, 2.5, -1.5, -2.5, -3.5}},
{{"Particle: 1"}, 256, 1, 9, 1, 1, 17179869184, {1.5, 2.5, 3.5, -0.5, -1.5, -2.5}},
{{"Particle: 2"}, 512, 2, 8, 4, 256, 34359738368, {2.5, 3.5, 4.5, 0.5, -0.5, -1.5}},
{{"Particle: 3"}, 768, 3, 7, 9, 6561, 51539607552, {3.5, 4.5, 5.5, 1.5, 0.5, -0.5}},
{{"Particle: 4"}, 1024, 4, 6, 16, 65536, 68719476736, {4.5, 5.5, 6.5, 2.5, 1.5, 0.5}},
{{"Particle: 5"}, 1280, 5, 5, 25, 390625, 85899345920, {5.5, 6.5, 7.5, 3.5, 2.5, 1.5}},
{{"Particle: 6"}, 1536, 6, 4, 36, 1679616, 103079215104, {6.5, 7.5, 8.5, 4.5, 3.5, 2.5}},
{{"Particle: 7"}, 1792, 7, 3, 49, 5764801, 120259084288, {7.5, 8.5, 9.5, 5.5, 4.5, 3.5}},
{{"Particle: 8"}, 2048, 8, 2, 64, 16777216, 137438953472, {8.5, 9.5, 10.5, 6.5, 5.5, 4.5}},
{{"Particle: 9"}, 2304, 9, 1, 81, 43046721, 154618822656, {9.5, 10.5, 11.5, 7.5, 6.5, 5.5}} ;
}
</pre>

===Reading the table with a C++ code ===
The code has been compiled and tested on GPC with the following modules :
<pre>
module load gcc/4.8.1 hdf5/187-v18-serial-gcc netcdf/4.1.3_hdf5_serial-gcc intel/14.0.0 python/2.7.2
gcc -I$SCINET_NETCDF_INC Test.cpp -o Test -lnetcdf_c++
</pre>
Test.cpp :
<pre>
#include <iostream>
#include <netcdfcpp.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

using namespace std;

#define FILE_NAME "particles.nc"
#define DIM_LEN 10 //number of records in file
int main(void)
{
typedef struct Particle
{
char name[16];
unsigned short int ADCcount;
int grid_i;
int grid_j;
float pressure;
double energy ;
long idnumber;
float pressure2[2][3];
} Particle;

//Definition of the data_in variable
Particle data_in[DIM_LEN];

//Initialization of the variable
for (int i=0; i<DIM_LEN; i++){
data_in[i].ADCcount=0;
data_in[i].grid_i=0;
data_in[i].grid_j=0;
data_in[i].pressure=0.;
data_in[i].energy=0.;
for (int j=0; j<2; j++){
for (int k=0; k<3; k++){
data_in[i].pressure2[j][k]=0.;
}
}
}

//Variables needed to open the NetCDF file :
int ncid, typeidd, varid;
int dimid;
int dimids[] = {0}, fieldid;

nc_def_dim(ncid, "NRecords", DIM_LEN, &dimid);
nc_def_var(ncid, "Data", typeidd, 1, dimids, &varid);

//Open the NetCDF file :
if (nc_open(FILE_NAME, NC_NOWRITE, &ncid)) cout<<"ERROR"<<endl; //NC_NOWRITE for read only, NC_WRITE for read and write
//Read the data and fill the variable data_in :
if (nc_get_var(ncid, varid, data_in)) cout<<"ERROR"<<endl;

for (int i=0; i<DIM_LEN; i++){
std::cout<<"ADCcount = "<<data_in[i].ADCcount
<<" ,idnumber = "<<data_in[i].idnumber
<<" ,grid_i = "<<data_in[i].grid_i
<<" ,grid_j = "<<data_in[i].grid_j
<<" ,pressure = "<<data_in[i].pressure
<<" ,energy = "<<data_in[i].energy
<<" ,name = "<<data_in[i].name
<<std::endl;
for(int j=0;j<2;j++){
std::cout<<data_in[i].pressure2[j][0]<<" "<<data_in[i].pressure2[j][1]<<" "<<data_in[i].pressure2[j][2]<<std::endl;
}
}
cout << "*** SUCCESS reading example file "<<FILE_NAME<<"!" << endl;
return 0;
}

</pre>

=== Using Parallel netCDF ===
The compound format (actually HDF5) isn't
compatible with parallel-netcdf. There are parallel I/O facilities in
NetCDF-4. To use the parallel features of netcdf-4, use the call
nc_open_par().

NetCDF table

2014-01-06T14:36:47Z

Brelier: /* Storing tables in NetCDF4 */

NetCDF table

2013-12-23T18:43:22Z

Brelier: /* Reading the table with a C++ code */