https://oldwiki.scinet.utoronto.ca/api.php?action=feedcontributions&feedformat=atom&user=Fertinazoldwiki.scinet.utoronto.ca - User contributions [en-gb]2026-04-12T22:23:20ZUser contributionsMediaWiki 1.35.12https://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9342BGQ OpenFOAM2018-05-14T20:09:12Z<p>Fertinaz: /* Wind Flow Around Building */</p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script by typing the modules you want load inside the batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations - Solves hanging problems<br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm. - Solves termination with signal 36 issue<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Typical OpenFOAM Applications on BG/Q ==<br />
A list of examples will be shared here. These sample cases are derived from applications that are run on BG/Q but changed for confidentiality reasons. It can guide new users for their specific use cases. Most of the information here is OpenFOAM specific, not BG/Q specific.<br />
<br />
=== Wind Flow Around Buildings ===<br />
This is a tutorial case that can be found in $FOAM_TUTORIALS/incompressible/simpleFoam/windAroundBuildings<br />
<br />
=== Rotational Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== LES Models in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== Multiphase Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can convert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells which allows users to fully utilize all 16 compute cores on a node.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9315BGQ OpenFOAM2018-05-04T20:20:24Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script by typing the modules you want load inside the batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations - Solves hanging problems<br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm. - Solves termination with signal 36 issue<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Typical OpenFOAM Applications on BG/Q ==<br />
A list of examples will be shared here. These sample cases are derived from applications that are run on BG/Q but changed for confidentiality reasons. It can guide new users for their specific use cases. Most of the information here is OpenFOAM specific, not BG/Q specific.<br />
<br />
=== Wind Flow Around Building ===<br />
This is a tutorial case that can be found in $FOAM_TUTORIALS/incompressible/simpleFoam/windAroundBuildings<br />
<br />
=== Rotational Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== LES Models in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== Multiphase Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can convert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells which allows users to fully utilize all 16 compute cores on a node.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9251BGQ OpenFOAM2018-05-03T02:48:38Z<p>Fertinaz: /* Typical OpenFOAM Application on BG/Q */</p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script by typing the modules you want load inside the batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations - Solves hanging problems<br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm. - Solves termination with signal 36 issue<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Typical OpenFOAM Applications on BG/Q ==<br />
A list of examples will be shared here. These sample cases are derived from applications that are run on BG/Q but changed for confidentiality reasons. It can guide new users for their specific use cases. Most of the information here is OpenFOAM specific, not BG/Q specific.<br />
<br />
=== Wind Flow Around Building ===<br />
This is a tutorial case that can be found in $FOAM_TUTORIALS/incompressible/simpleFoam/windAroundBuildings<br />
<br />
=== MRF Type Solvers in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== LES Models in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== Multiphase Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can convert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells which allows users to fully utilize all 16 compute cores on a node.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9250BGQ OpenFOAM2018-05-02T16:22:34Z<p>Fertinaz: /* Using OpenFOAM on BG/Q */</p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script by typing the modules you want load inside the batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations - Solves hanging problems<br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm. - Solves termination with signal 36 issue<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Typical OpenFOAM Application on BG/Q ==<br />
A list of examples will be shared here. These sample cases are derived from applications that are run on BG/Q but changed for confidentiality reasons. It can guide new users for their specific use cases. Most of the information here is OpenFOAM specific, not BG/Q specific.<br />
<br />
=== Flow Around Solid Regions ===<br />
Information will be added soon!<br />
<br />
=== MRF Type Solvers in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== LES Models in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== Multiphase Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can convert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells which allows users to fully utilize all 16 compute cores on a node.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9248BGQ OpenFOAM2018-05-01T16:37:06Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations - Solves hanging problems<br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm. - Solves termination with signal 36 issue<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Typical OpenFOAM Application on BG/Q ==<br />
A list of examples will be shared here. These sample cases are derived from applications that are run on BG/Q but changed for confidentiality reasons. It can guide new users for their specific use cases. Most of the information here is OpenFOAM specific, not BG/Q specific.<br />
<br />
=== Flow Around Solid Regions ===<br />
Information will be added soon!<br />
<br />
=== MRF Type Solvers in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== LES Models in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
=== Multiphase Flows in OpenFOAM ===<br />
Information will be added soon!<br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can convert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells which allows users to fully utilize all 16 compute cores on a node.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9247BGQ OpenFOAM2018-05-01T16:29:38Z<p>Fertinaz: /* Running Parallel Meshing */</p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations - Solves hanging problems<br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm. - Solves termination with signal 36 issue<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can covert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ&diff=9246BGQ2018-05-01T16:27:45Z<p>Fertinaz: </p>
<hr />
<div>{{Infobox Computer<br />
|image=[[Image:Blue_Gene_Cabinet.jpeg|center|300px|thumb]]<br />
|name=Blue Gene/Q (BGQ)<br />
|installed=Aug 2012, Nov 2014<br />
|operatingsystem= RH6.3, CNK (Linux) <br />
|loginnode= bgqdev-fen1<br />
|nnodes= 4096 nodes (65,536 cores)<br />
|rampernode=16 GB <br />
|corespernode=16 (64 threads)<br />
|interconnect=5D Torus (jobs), QDR Infiniband (I/O) <br />
|vendorcompilers= bgxlc, bgxlf<br />
|queuetype=Loadleveler<br />
}}<br />
<br />
==System Status==<br />
<br />
The current BGQ system status can be found on the wiki's [[Main Page]].<br />
<br />
==SOSCIP & LKSAVI==<br />
<br />
The BGQ is a Southern Ontario Smart Computing<br />
Innovation Platform ([http://soscip.org/ SOSCIP]) BlueGene/Q supercomputer located at the<br />
University of Toronto's SciNet HPC facility. The SOSCIP <br />
multi-university/industry consortium is funded by the Ontario Government <br />
and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].<br />
<br />
A half-rack of BlueGene/Q (8,192 cores) was purchased by the [http://likashingvirology.med.ualberta.ca/ Li Ka Shing Institute of Virology] at the University of Alberta in late fall 2014 and integrated into the existing BGQ system.<br />
<br />
The combined 4 rack system is the fastest Canadian supercomputer on the [http://top500.org/ top 500], currently at the 120th place (Nov 2015).<br />
<br />
== Support Email ==<br />
<br />
Please use [mailto:bgq-support@scinet.utoronto.ca <bgq-support@scinet.utoronto.ca>] for BGQ-specific inquiries.<br />
<br />
==Specifications==<br />
<br />
BGQ is an extremely dense and energy efficient 3rd generation Blue Gene IBM supercomputer built around a system-on-a-chip compute node that has a 16core 1.6GHz PowerPC based CPU (PowerPC A2) with 16GB of Ram. The nodes are bundled in groups of 32 into a node board (512 cores), and 16 boards make up a midplane (8192 cores) with 2 midplanes per rack, or 16,348 cores and 16 TB of RAM per rack. The compute nodes run a very lightweight Linux-based operating system called CNK ('''C'''ompute '''N'''ode '''K'''ernel). The compute nodes are all connected together using a custom 5D torus highspeed interconnect. Each rack has 16 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the filesystem. SciNet's BGQ consists of 8 mdiplanes (four-racks) totalling 65,536 cores and 64TB of RAM.<br />
<br />
[[Image:BlueGeneQHardware2.png‎ |center]]<br />
<br />
=== 5D Torus Network ===<br />
<br />
The network topology of BlueGene/Q is a five-dimensional (5D) torus, with direct links between the nearest neighbors in the ±A, ±B, ±C, ±D, and ±E directions. As such there are only a few optimum block sizes that will use the network efficiently.<br />
<br />
{|border="1" cellspacing="0" cellpadding="2"<br />
| '''Node Boards '''<br />
| '''Compute Nodes'''<br />
| '''Cores'''<br />
| '''Torus Dimensions'''<br />
|-<br />
| 1<br />
| 32<br />
| 512<br />
| 2x2x2x2x2<br />
|-<br />
| 2 (adjacent pairs)<br />
| 64<br />
| 1024<br />
| 2x2x4x2x2<br />
|-<br />
| 4 (quadrants)<br />
| 128<br />
| 2048<br />
| 2x2x4x4x2<br />
|-<br />
| 8 (halves)<br />
| 256<br />
| 4096<br />
| 4x2x4x4x2<br />
|-<br />
| 16 (midplane)<br />
| 512<br />
| 8192<br />
| 4x4x4x4x2<br />
|-<br />
| 32 (1 rack)<br />
| 1024<br />
| 16384<br />
| 4x4x4x8x2 <br />
|-<br />
| 64 (2 racks)<br />
| 2048<br />
| 32768<br />
| 4x4x8x8x2<br />
|-<br />
| 96 (3 racks)<br />
| 3072<br />
| 49152<br />
| 4x4x12x8x2<br />
|-<br />
| 128 (4 racks)<br />
| 4096<br />
| 65536<br />
| 8x4x8x8x2<br />
|}<br />
<br />
== Login/Devel Node ==<br />
<br />
The development node is '''bgqdev-fen1''' which one can login to from the regular '''login.scinet.utoronto.ca''' login nodes or directly from outside using '''bgqdev.scinet.utoronto.ca''', e.g.<br />
<pre><br />
$ ssh -l USERNAME bgqdev.scinet.utoronto.ca -X<br />
</pre><br />
where USERNAME is your username on the BGQ and the <tt>-X</tt> flag is optional, needed only if you will use X graphics.<br/><br />
Note: To learn how to setup ssh keys for logging in please see [[Ssh keys]].<br />
<br />
These development node is a Power7 machines running Linux which serve as the compilation and submission host for the BGQ. Programs are cross-compiled for the BGQ on this node and then submitted to the queue using loadleveler.<br />
<br />
===Modules and Environment Variables===<br />
<br />
To use most packages on the SciNet machines - including most of the compilers - , you will have to use the `modules' command. The command <tt>module load some-package</tt> will set your environment variables (<tt>PATH</tt>, <tt>LD_LIBRARY_PATH</tt>, etc) to include the default version of that package. <tt>module load some-package/specific-version</tt> will load a specific version of that package. This makes it very easy for different users to use different versions of compilers, MPI versions, libraries etc.<br />
<br />
A list of the installed software can be seen on the system by typing <br />
<pre><br />
$ module avail<br />
</pre><br />
<br />
To load a module (for example, the default version of the intel compilers)<br />
<pre><br />
$ module load vacpp<br />
</pre><br />
To unload a module<br />
<pre><br />
$ module unload vacpp<br />
</pre><br />
To unload all modules<br />
<pre><br />
$ module purge<br />
</pre><br />
<br />
These commands can go in your .bashrc files to make sure you are using the correct packages.<br />
<br />
Modules that load libraries, define environment variables pointing to the location of library files and include files for use Makefiles. These environment variables follow the naming convention<br />
<pre><br />
$SCINET_[short-module-name]_BASE<br />
$SCINET_[short-module-name]_LIB<br />
$SCINET_[short-module-name]_INC<br />
</pre><br />
for the base location of the module's files, the location of the libraries binaries and the header files, respectively.<br />
<br />
So to compile and link the library, you will have to add <tt>-I${SCINET_[module-basename]_INC}</tt> and <tt>-L${SCINET_[module-basename]_LIB}</tt>, respectively, in addition to the usual <tt>-l[libname]</tt>. <br />
<br />
Note that a <tt>module load</tt> command ''only'' sets the environment variables in your current shell (and any subprocesses that the shell launches). It does ''not'' effect other shell environments.<br />
<br />
If you always require the same modules, it is easiest to load those modules in your <tt>.bashrc</tt> and then they will always be present in your environment; if you routinely have to flip back and forth between modules, it is easiest to have almost no modules loaded in your <tt>.bashrc</tt> and simply load them as you need them (and have the required <tt>module load</tt> commands in your job submission scripts).<br />
<br />
=== Compilers ===<br />
<br />
The BGQ uses IBM XL compilers to cross-compile code for the BGQ. Compilers are available for FORTRAN, C, and C++. They are accessible by default, or by loading the '''xlf''' and '''vacpp''' modules. The compilers by default produce<br />
static binaries, however with BGQ it is possible to now use dynamic libraries as well. The compilers follow the XL conventions with the prefix '''bg''',<br />
so '''bgxlc''' and '''bgxlf90''' are the C and FORTRAN compilers respectively. <br />
<br />
Most users however will use the MPI variants, i.e. '''mpixlf90''' and '''mpixlc''' and which are available by loading<br />
the '''mpich2''' module. <br />
<pre><br />
module load mpich2<br />
</pre><br />
<br />
It is recommended to use at least the following flags when compiling and linking<br />
<pre><br />
-O3 -qarch=qp -qtune=qp<br />
</pre><br />
<br />
If you want to build a package for which the configure script tries to run small test jobs, the cross-compiling nature of the bgq can get in the way. In that case, you should use the interactive [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] environment as described below.<br />
<br />
== ION/Devel Nodes ==<br />
<br />
There are also bgq native development nodes named '''bgqdev-ion[01-24]''' which one can login to directly, i.e. ssh, from '''bgqdev-fen1'''. These nodes are extra I/O nodes that are essentially the same as the BGQ compute nodes with the exception that they run a full RedHat Linux and have an infiniband interface providing direct network access. Unlike the regular development node, '''bgqdev-fen1''', which is Power7, this node has the same BGQ A2 processor, and thus cross compilations are not required which can make building some software easier. <br />
<br />
'''NOTE''': BGQ MPI jobs can be compiled on these nodes, however can not be run locally as the mpich2 is setup for the BGQ network and thus will fail on these nodes.<br />
<br />
== Job Submission ==<br />
<br />
As the BlueGene/Q architecture is different from the development nodes, you cannot run applications intended/compiled for the BGQ on the devel nodes. The only way to run (or even test) your program is to submit a job to the BGQ. Jobs are submitted as scripts through loadleveler. That script must then use '''runjob''' to start the job, which in many ways similar to mpirun or mpiexec. As shown above in the network topology overview, there are only a few optimum job size configurations which is also further constrained by each block requiring a minimum of one IO node. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest block size. Normally a block size matches the job size to offer fully dedicated resources to the job. Smaller jobs can be run within the same block however this results in shared resources (network and IO) and are referred to as sub-block jobs and are described in more detail below. <br />
<br />
=== runjob ===<br />
<br />
All BGQ runs are launched using '''runjob''' which for those familiar with MPI is analogous to mpirun/mpiexec. Jobs run on a block, which is a predefined group of nodes that have already been configured and booted. There are two ways to get a block. One way is to use a 30-minute 'debugjob' session (more about that below). The other, more common case, is using a job script submitted and are running using loadleveler. Inside the job script, this block is set for you, and you do not have to specify the block name. For example, if your loadleveler job script requests 64 nodes, each with 16 cores (for a total of 1024 cores), from within that job script, you can run a job with 16 processes per node and 1024 total processes with<br />
<pre><br />
runjob --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in<br />
</pre><br />
Here, <tt>--np 1024</tt> sets the total number of mpi tasks, while <tt>--ranks-per-node=16</tt> specifies that 16 processes should run on each node.<br />
For pure mpi jobs, it is advisable always to give the number of ranks per node, because the default value of 1 may leave 15 cores on the node idle. The argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. <br />
<br />
<!-- (Note: If this were not a loadleveler job, and the block ID was R00-M0-N03-64, the command would be "<tt>runjob --block R00-M0-N03-64 --np 1024 --ranks-per-node=16 --cwd=$PWD : $PWD/code -f file.in</tt>") --><br />
<br />
runjob flags are shown with<br />
<pre><br />
runjob -h<br />
</pre><br />
<br />
a particularly useful one is<br />
<br />
<pre><br />
--verbose #<br />
</pre><br />
<br />
where # is from 1-7 which can be helpful in debugging an application.<br />
<br />
=== How to set ranks-per-node ===<br />
<br />
There are 16 cores per node, but the argument to ranks-per-node may be 1, 2, 4, 8, 16, 32, or 64. While it may seem natural to set ranks-per-node to 16, this is not generally recommended. On the BGQ, one can efficiently run more than 1 process per core, because each core has four "hardware threads" (similar to HyperThreading on the GPC and Simultaneous Multi Threading on the TCS and P7), which can keep the different parts of each core busy at the same time. One would therefore ideally use 64 ranks per node. There are two main reason why one might not set ranks-per-node to 64:<br />
# The memory requirements do not allow 64 ranks (each rank only has 256MB of memory)<br />
# The application is more efficient in a hybrid MPI/OpenMP mode (or MPI/pthreads). Using less ranks-per-node, the hardware threads are used as OpenMP threads within each process.<br />
Because threads can share memory, the memory requirements of the hybrid runs is typically smaller than that of pure MPI runs.<br />
<br />
Note that the total number of mpi processes in a runjob (i.e., the --np argument) should be the ranks-per-node times the number of nodes (set by bg_size in the loadleveler script). So for the same number of nodes, if you change ranks-per-node by a factor of two, you should also multiply the total number of mpi processes by two.<br />
<br />
=== Queue Limits ===<br />
<br />
The maximum wall_clock_limit is 24 hours. Official SOSCIP project jobs are prioritized over all other jobs using a fairshare algorithm with a 14 day rolling window.<br />
<br />
A 64 node block is reserved for development and interactive testing for 16 hours, from 8AM to midnight, everyday including weekends. While you can still reserve an interactive block from midnight to 8AM, the priority is given to batch jobs at that time interval in order to keep the machine usage as high as possible. This block is accessed by using the [[BGQ#Interactive_Use_.2F_Debugging | <tt>'''debugjob'''</tt>]] command which has a 30 minute maximum wall_clock_limit. The purpose of this reservation is to ensure short testing jobs are run quickly without being held up by longer production type jobs.<br />
<br />
<!-- We need to recover this functionality again. At the moment it doesn't work<br />
=== BACKFILL scheduling ===<br />
To optimize the cluster usage, we encourage users to submit jobs according to the available resources on BGQ. The command <span style="color: red;font-weight: bold;">llAvailableResources</span> gives for example :<br />
<pre><br />
On the Devel system : only a debugjob can start immediately<br />
<br />
On the Prod. system : a job will start immediately if you use 512 nodes requesting a walltime T <= 21 hours and 11 min <br />
On the Prod. system : a job will start immediately if you use 256 nodes requesting a walltime T <= 21 hours and 11 min <br />
On the Prod. system : a job will start immediately if you use 128 nodes requesting a walltime T <= 24 hours and 0 min <br />
On the Prod. system : a job will start immediately if you use 64 nodes requesting a walltime T <= 24 hours and 0 min<br />
</pre><br />
--><br />
<br />
=== Batch Jobs ===<br />
<br />
Job submission is done through loadleveler with a few blue gene specific commands. The command "bg_size" is in number of nodes, not cores, so a bg_size=64 would be 64x16=1024 cores.<br />
<br />
The parameter <span style="font-weight: bold;">bg_size</span> can only be equal to 64, 128, 256, 512, 1024 and 2048.<br />
<br />
<span style="font-weight: bold;">np</span> &le; ranks-per-node * bg_size<br />
<br />
ranks-per-node &le; np<br />
<br />
(ranks-per-node * OMP_NUM_THREADS ) &le; 64 <br />
<br />
np : number of MPI processes<br />
<br />
ranks-per-node : number of MPI processes per node = 1 , 2 , 4 , 8 , 16 , 32 , 64<br />
<br />
OMP_NUM_THREADS : number of OpenMP thread per MPI process (for hybrid codes) = 1 , 2 , 4 , 8 , 16 , 32 , 64<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgsample<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Launch all BGQ jobs using runjob<br />
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags<br />
</pre><br />
<br />
To submit to the queue use <br />
<br />
<pre><br />
llsubmit myscript.sh<br />
</pre><br />
=== Steps ( Job dependency) ===<br />
LoadLeveler has a lot of advanced features to control job submission and execution. One of these features is called steps. This feature allows a series of jobs to be submitted using one script with dependencies defined between the jobs. What this allows is for a series of jobs to be run sequentially, waiting for the previous job, called a step, to be finished before the next job is started. The following example uses the same LoadLeveler script as previously shown, however the #@ step_name and #@ dependency directives are used to rerun the same case three times in a row, waiting until each job is finished to start the next.<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgsample<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ step_name = step1 <br />
# @ queue<br />
# Launch the first step :<br />
if [ $LOADL_STEP_NAME = "step1" ]; then<br />
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags<br />
fi<br />
<br />
# @ job_name = bgsample<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ step_name = step2 <br />
# @ dependency = step1 == 0 <br />
# @ queue<br />
# Launch the second step if the first one has returned 0 (done successfully) :<br />
if [ $LOADL_STEP_NAME = "step2" ]; then<br />
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags<br />
fi<br />
<br />
# @ job_name = bgsample<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ step_name = step3 <br />
# @ dependency = step2 == 0 <br />
# @ queue<br />
# Launch the third step if the second one has returned 0 (done successfully) :<br />
if [ $LOADL_STEP_NAME = "step3" ]; then<br />
runjob --np 1024 --ranks-per-node=16 --envs OMP_NUM_THREADS=1 --cwd=$SCRATCH/ : $HOME/mycode.exe myflags<br />
fi<br />
<br />
</pre><br />
<br />
=== Monitoring Jobs ===<br />
<br />
To see running jobs<br />
<br />
<pre><br />
llq2<br />
</pre><br />
<br />
or<br />
<br />
<pre><br />
llq -b<br />
</pre><br />
<br />
<br />
to cancel a job use<br />
<br />
<pre><br />
llcancel JOBID<br />
</pre><br />
<br />
and to look at details of the bluegene resources use<br />
<br />
<pre><br />
llbgstatus -M all<br />
</pre><br />
<br />
'''Note: the loadleveler script commands are not run on a bgq compute node but on the front-end node. Only programs started with runjob run on the bgq compute nodes. You should therefore keep scripting in the submission script to a bare minimum.'''<br />
<br />
=== Monitoring Stats ===<br />
<br />
Use llbgstats to monitor your own stats and/or your group stats. PIs can also print their (current) monthly report.<br />
<pre><br />
llbgstats -h<br />
</pre><br />
<br />
=== Interactive Use / Debugging ===<br />
<br />
As BGQ codes are cross-compiled they cannot be run direclty on the front-nodes. <br />
Users however only have access to the BGQ through loadleveler which is appropriate for batch jobs, <br />
however an interactive session is typically beneficial when debugging and developing. As such a <br />
script has been written to allow a session in which runjob can be run interactively. The script<br />
uses loadleveler to setup a block and set all the correct environment variables and then launch a spawned shell on<br />
the front-end node. The '''debugjob''' session currently allows a 30 minute session on 64 nodes and when run on <br />
'''<tt>bgqdev</tt>''' runs in a dedicated reservation as described previously in the [[BGQ#Queue_Limits | queue limits]] section. <br />
<br />
<pre><br />
[user@bgqdev-fen1]$ debugjob<br />
<br />
[user@bgqdev-fen1]$ runjob --np 64 --ranks-per-node=16 --cwd=$PWD : $PWD/my_code -f myflags<br />
<br />
[user@bgqdev-fen1]$ exit<br />
</pre><br />
<br />
For debugging, gdb and Allinea DDT are available. The latter is recommended as it automatically attaches to all the processes of a process (instead of attaching a gdbtool by hand (as explained in the BGQ Application Development guide, link below). Simply compile with <tt>-g</tt>, load the <tt>ddt/4.1</tt> module, type <tt>ddt</tt> and follow the graphical user interface. The DDT user guide can be found below.<br />
<br />
Note: when running a job under ddt, you'll need to add "<tt>--ranks-per-node=X</tt>" to the "runjob arguments" field.<br />
<br />
Apart from debugging, this environment is also useful for building libraries and applications that need to run small tests as part of their 'configure' step. Within the debugjob session, applications compiled with the bgxl compilers or the mpcc/mpCC/mpfort wrappers, will automatically run on the BGQ, skipping the need for the runjob command, provided if you set the following environment variables <br />
<pre><br />
$ export BG_PGM_LAUNCHER=yes<br />
$ export RUNJOB_NP=1<br />
</pre><br />
The latter setting sets the number of mpi processes to run. Most configure scripts expect only one mpi process, thus, <tt>RUNJOB_NP=1</tt> is appropriate.<br />
<br />
<br />
debugjob session with an executable implicitly calls runjob with 1 mpi task :<br />
<pre><br />
debugjob -i<br />
**********************************************************<br />
Interactive BGQ runjob shell using bgq-fen1-ib0.10295.0 and <br />
LL14040718574824 for 30 minutes with 64 NODES (1024 cores). <br />
IMPLICIT MODE: running an executable implicitly calls runjob<br />
with 1 mpi task<br />
Exit shell when finished. <br />
**********************************************************<br />
</pre><br />
<br />
=== Sub-block jobs ===<br />
<br />
BGQ allows multiple applications to share the same block, which is referred to as sub-block jobs, however this needs to be done from within the same loadleveler submission script using multiple calls to runjob. To run a sub-block job, you need to specify a "--corner" within the block to start each job and a 5D Torus AxBxCxDxE "--shape". The starting corner will depend on the specific block details provided by loadleveler and the shape and size of job trying to be used. <br />
<br />
Figuring out what the corners and shapes should be is very tricky (especially since it depends on the block you get allocated). For that reason, we've created a script called <tt>subblocks</tt> that determines the corners and shape of the sub-blocks. It only handles the (presumable common) case in which you want to subdivide the block into n equally sized sub-blocks, where n may be 1,2,4,8,16 and 32.<br />
<br />
Here is an example script calling <tt>subblocks</tt> with a size of 4 that will return the appropriate $SHAPE argument and an array of 16 starting $CORNER. <br />
<source lang="bash"><br />
#!/bin/bash<br />
# @ job_name = bgsubblock<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job SUBBLOCK "<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue<br />
<br />
# Using subblocks script to set $SHAPE and array of ${CORNERS[n]}<br />
# with size of subblocks in nodes (ie similiar to bg_size)<br />
<br />
# In this case 16 sub-blocks of 4 cnodes each (64 total ie bg_size)<br />
source subblocks 4<br />
<br />
# 16 jobs of 4 each<br />
for (( i=0; i < 16 ; i++)); do<br />
runjob --corner ${CORNER[$i]} --shape $SHAPE --np 64 --ranks-per-node=16 : your_code_here > $i.out &<br />
done<br />
wait<br />
</source><br />
Remember that subjobs are not the ideal way to run on the BlueGene/Qs. One needs to consider that these sub-blocks all have to share the same I/O nodes, so for I/O intensive jobs this will be an inefficient setup. Also consider that if you need to run such small jobs that you have to run in sub-blocks, it may be more efficient to use other clusters such as the GPC.<br />
<br />
Let us know if you run into any issues with this technique, please contact bgq-support for help.<br />
<br />
== Filesystem ==<br />
<br />
The BGQ has its own dedicated 500TB file system based on GPFS (General Parallel File System). There are two main systems for user data: /home, a small, backed-up space where user home directories are located, and /scratch, a large system for input or output data for jobs; data on /scratch is not backed up. The path to your home directory is in the environment variable $HOME, and will look like /home/G/GROUP/USER, . The path to your scratch directory is in the environment variable $SCRATCH, and will look like /scratch/G/GROUP/USER (following the conventions of the rest of the SciNet systems). <br />
<br />
{|border="1" cellpadding="10" cellspacing="0"<br />
|-<br />
! {{Hl2}} | file system <br />
! {{Hl2}} | purpose <br />
! {{Hl2}} | user quota <br />
! {{Hl2}} | backed up<br />
! {{Hl2}} | purged<br />
|- <br />
| /home<br />
| development<br />
| 50 GB<br />
| yes<br />
| never<br />
|-<br />
| /scratch<br />
| computation<br />
| first of (20 TB ; 1 million files)<br />
| no<br />
| not currently<br />
|}<br />
<br />
===Transfering files===<br />
The BGQ GPFS file system, except for HPSS, is '''not''' shared with the other SciNet systems (gpc, tcs, p7, arc), nor is the other file system mounted on the BGQ. <br />
Use scp to copy files from one file system to the other, e.g., from bgqdev-fen1, you could do<br />
<pre><br />
$ scp -c arcfour login.scinet.utoronto.ca:code.tgz .<br />
</pre><br />
or from a login node you could do<br />
<pre><br />
$ scp -c arcfour code.tgz bgqdev.scinet.utoronto.ca:<br />
</pre><br />
The flag <tt>-c arcfour</tt> is optional. It tells scp (or really, ssh), to use a non-default encryption. The one chosen here, arcfour, has been found to speed up the transfer by a factor of two (you may expect around 85MB/s). This encryption method is only recommended for copying from the BGQ file system to the regular SciNet GPFS file system or back. <br />
<br />
Note that although these transfers are witihin the same data center, you have to use the full names of the systems, login.scinet.utoronto.ca and bgq.scinet.utoronto.ca, respectively, and that you will be asked you for your password.<br />
<br />
===How much Disk Space Do I have left?===<br />
<br />
The <tt>'''diskUsage'''</tt> command, available on the bgqdev nodes, provides information in a number of ways on the home and scratch file systems. For instance, how much disk space is being used by yourself and your group (with the -a option), or how much your usage has changed over a certain period ("delta information") or you may generate plots of your usage over time. Please see the usage help below for more details.<br />
<pre><br />
Usage: diskUsage [-h|-?| [-a] [-u <user>] [-de|-plot]<br />
-h|-?: help<br />
-a: list usages of all members on the group<br />
-u <user>: as another user on your group<br />
-de: include delta information<br />
-plot: create plots of disk usages<br />
</pre><br />
Note that the information on usage and quota is only updated hourly!<br />
<br />
===Bridge to HPSS===<br />
<br />
BGQ users may transfer material to/from HPSS via the GPC archive queue. On the HPSS gateway node (gpc-archive01), the BGQ GPFS file systems are mounted under a single mounting point /bgq (/bgq/scratch and /bgq/home). For detailed information on the use of HPSS [https://support.scinet.utoronto.ca/wiki/index.php/HPSS please read the HPSS wiki section.]<br />
<br />
== Software modules installed on the BGQ ==<br />
<br />
{| border="1" cellpadding="10" cellspacing="0"<br />
!{{Hl2}} |Software <br />
!{{Hl2}}| Version<br />
!{{Hl2}}| Comments<br />
!{{Hl2}}| Command/Library<br />
!{{Hl2}}| Module Name<br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Compilers & Development Tools'''''<br />
|-<br />
|IBM fortran compiler<br />
|14.1<br />
|These are cross compilers<br />
|<tt>bgxlf,bgxlf_r,bgxlf90,...</tt><br />
|xlf<br />
|-<br />
|IBM c/c++ compilers<br />
|12.1<br />
|These are cross compilers<br />
|<tt>bgxlc,bgxlC,bgxlc_r,bgxlC_r,...</tt><br />
|vacpp<br />
|-<br />
|MPICH2 MPI library<br />
|1.4.1<br />
|There are 4 versions (see BGQ Applications Development document).<br />
|<tt>mpicc,mpicxx,mpif77,mpif90</tt><br />
|mpich2<br />
|- <br />
| GCC Compiler<br />
| 4.4.6, 4.8.1<br />
| GNU Compiler Collection for BGQ<br>(4.8.1 requires binutils/2.23 to be loaded)<br />
| <tt>powerpc64-bgq-linux-gcc, powerpc64-bgq-linux-g++, powerpc64-bgq-linux-gfortran</tt><br />
| <tt>bgqgcc</tt><br />
|-<br />
| Clang Compiler<br />
| r217688-20140912, r263698-20160317<br />
| Clang cross-compilers for bgq<br />
| <tt>powerpc64-bgq-linux-clang, powerpc64-bgq-linux-clang++</tt><br />
| <tt>bgclang</tt><br />
|-<br />
| Binutils<br />
| 2.21.1, 2.23<br />
| Cross-compilation utilities<br />
| <tt>addr2line, ar, ld, ...</tt><br />
| <tt>binutils</tt><br />
|-<br />
| CMake <br />
| 2.8.8, 2.8.12.1<br />
| cross-platform, open-source build system<br />
| <tt>cmake</tt><br />
| <tt>cmake</tt><br />
|-<br />
| Git<br />
| 1.9.5<br />
| Revision control system<br />
| <tt>git, gitk</tt><br />
| <tt>git</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Debug/performance tools'''''<br />
|-<br />
| [https://www.gnu.org/software/gdb/ gdb]<br />
| 7.2<br />
| GNU Debugger<br />
| <tt>gdb</tt><br />
| <tt>gdb</tt><br />
|-<br />
| [https://www.gnu.org/software/ddd/ ddd]<br />
| 3.3.12<br />
| GNO Data Display Debugger<br />
| <tt>ddd</tt><br />
| <tt>ddd</tt><br />
|- <br />
| [http://www.allinea.com/products/ddt/ DDT]<br />
| 4.1, 4.2, 5.0.1<br />
| Allinea's Distributed Debugging Tool<br />
| <tt>ddt</tt><br />
| <tt>ddt</tt><br />
|- <br />
| [[HPCTW]]<br />
| 1.0<br />
| BGQ MPI and Hardware Counters<br />
| <tt>libmpihpm.a, libmpihpm_smp.a, libmpitrace.a </tt><br />
| <tt>hptibm</tt><br />
|- <br />
| [[MemP]]<br />
| 1.0.3<br />
| BGQ Memory Stats<br />
| <tt>libmemP.a </tt><br />
| <tt>memP</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Storage tools/libraries'''''<br />
|-<br />
| HDF5<br />
| 1.8.9-v18<br />
| Scientific data storage and retrieval<br />
| <tt>h5ls, h5diff, ..., libhdf5</tt><br />
| <tt>hdf5/189-v18-serial-xlc*<br/>hdf5/189-v18-mpich2-xlc</tt><br />
|-<br />
| HDF5<br />
| 1.8.12-v18<br />
| Scientific data storage and retrieval<br />
| <tt>h5ls, h5diff, ..., libhdf5</tt><br />
| <tt>hdf5/1812-v18-serial-gcc<br/>hdf5/1812-v18-mpich2-gcc</tt><br />
|-<br />
| NetCDF<br />
| 4.2.1.1<br />
| Scientific data storage and retrieval<br />
| <tt>ncdump,ncgen,libnetcdf</tt><br />
| <tt>netcdf/4.2.1.1-serial-xlc*<br/>netcdf/4.2.1.1-mpich2-xlc</tt><br />
|-<br />
| Parallel NetCDF<br />
| 1.3.1<br />
| Parallel scientific data storage and retrieval using MPI-IO<br />
| <tt>libpnetcdf.a</tt><br />
| <tt>parallel-netcdf</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Libraries'''''<br />
|-<br />
| ESSL<br />
| 5.1<br />
| IBM Engineering and Scientific Subroutine Library (manual below)<br />
| <tt>libesslbg,libesslsmpbg</tt><br />
| <tt>essl</tt><br />
|-<br />
| WSMP<br />
| 15.06.01<br />
| Watson Sparse Matrix Package<br />
| <tt>libpwsmpBGQ.a</tt><br />
| <tt>WSMP</tt><br />
|- <br />
| FFTW<br />
| 2.1.5, 3.3.2, 3.1.2-esslwrapper<br />
| Fast fourier transform <br />
| <tt>libsfftw,libdfftw,libfftw3, libfftw3f</tt><br />
| <tt>fftw/2.1.5, fftw/3.3.2, fftw/3.1.2-esslwrapper</tt><br />
|-<br />
| LAPACK + ScaLAPACK<br />
| 3.4.2 + 2.0.2<br />
| Linear algebra routines. A subset of Lapack may be found in ESSL as well.<br />
| <tt>liblapack, libscalpack</tt><br />
| lapack<br />
|-<br />
| GSL<br />
| 1.15<br />
| GNU Scientific Library<br />
| <tt>libgsl, libgslcblas</tt><br />
| <tt>gsl</tt><br />
|-<br />
| BOOST<br />
| 1.47.0, 1.54, 1.57<br />
| C++ Boost libraries<br />
| <tt>libboost...</tt><br />
| <tt>cxxlibraries/boost</tt><br />
|-<br />
| bzip2 + szip + zlib<br />
| 1.0.6 + 2.1 + 1.2.7<br />
| compression libraries<br />
| <tt>libbz2,libz,libsz</tt><br />
| <tt>compression</tt><br />
|-<br />
| METIS<br />
| 5.0.2<br />
| Serial Graph Partitioning and Fill-reducing Matrix Ordering<br />
| <tt>libmetis</tt><br />
| <tt>metis</tt><br />
|-<br />
| ParMETIS<br />
| 4.0.2<br />
| Parallel graph partitioning and fill-reducing matrix ordering<br />
| <tt>libparmetis</tt><br />
| <tt>parmetis</tt> <br />
|-<br />
| OpenSSL<br />
| 1.0.2 <br />
| General-purpose cryptography library<br />
| <tt>libcrypto, libssl</tt><br />
| <tt>openssl</tt> <br />
|-<br />
| FILTLAN<br />
| 1.0<br />
| The Filtered Lanczos Package <br />
| <tt>libdfiltlan,libdmatkit,libsfiltlan,libsmatkit</tt><br />
| <tt>FILTLAN</tt> <br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Scripting/interpreted languages'''''<br />
|-<br />
| [[Python]]<br />
| 2.6.6<br />
| Python programming language<br />
| <tt>/bgsys/tools/Python-2.6/bin/python</tt><br />
| <tt>python</tt><br />
|-<br />
| [[Python]]<br />
| 2.7.3<br />
| Python programming language. Modules included : numpy-1.8.0, pyFFTW-0.9.2, astropy-0.3, scipy-0.13.3, mpi4py-1.3.1, h5py-2.2.1<br />
| <tt>/scinet/bgq/tools/Python/python2.7.3-20131205/bin/python</tt><br />
| <tt>python</tt><br />
|-<br />
| [[Python]]<br />
| 3.2.2<br />
| Python programming language<br />
| <tt>/bgsys/tools/Python-3.2/bin/python3</tt><br />
| <tt>python</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Applications'''''<br />
|-<br />
| [http://www.abinit.org/ ABINIT]<br />
| 7.10.4<br />
| An atomic-scale simulation software suite<br />
| <tt>abinit</tt><br />
| <tt>abinit</tt><br />
|-<br />
| [http://www.berkeleygw.org/ BerkeleyGW library]<br />
| 1.0.4-2.0.0436<br />
| Computes quasiparticle properties and the optical responses of a large variety of materials<br />
| <tt>libBGW_wfn.a, wfn_rho_vxc_io_m.mod</tt><br />
| <tt>BGW-paratec</tt><br />
|-<br />
| [https://www.cp2k.org/ CP2K]<br />
| 2.3, 2.4, 2.5.1, 2.6.1<br />
| DFT molecular dynamics, MPI <br />
| <tt>cp2k.psmp</tt><br />
| <tt>cp2k</tt><br />
|-<br />
| [http://www.cpmd.org/ CPMD]<br />
| 3.15.3, 3.17.1<br />
| Carr-Parinello molecular dynamics, MPI<br />
| <tt>cpmd.x</tt><br />
| <tt>cpmd</tt><br />
|-<br />
| gnuplot<br />
| 4.6.1<br />
| interactive plotting program to be run on front-end nodes<br />
| <tt>gnuplot</tt><br />
| <tt>gnuplot</tt><br />
|-<br />
| LAMMPS<br />
| Nov 2012/7Dec15/7Dec15-mpi<br />
| Molecular Dynamics <br />
| <tt>lmp_bgq</tt><br />
| <tt>lammps</tt><br />
|-<br />
| NAMD<br />
| 2.9<br />
| Molecular Dynamics <br />
| <tt>namd2</tt><br />
| <tt>namd/2.9-smp</tt><br />
|-<br />
| [http://www.quantum-espresso.org/index.php Quantum Espresso]<br />
| 5.0.3/5.2.1<br />
| Molecular Structure / Quantum Chemistry <br />
| <tt>qe_pw.x, etc</tt><br />
| <tt>espresso</tt><br />
|-<br />
| [[BGQ_OpenFOAM | OpenFOAM]]<br />
| 2.2.0, 2.3.0, 2.4.0, 3.0.1<br />
| Computational Fluid Dynamics<br />
| <tt>icofoam,etc. </tt><br />
| <tt>openfoam/2.2.0, openfoam/2.3.0, openfoam/2.4.0</tt><br />
|-<br />
|colspan=5 style='background: #E0E0E0'|'''''Beta Tests'''''<br />
|-<br />
| WATSON API<br />
| beta<br />
| Natural Language Processing<br />
| <tt>watson_beta</tt><br />
| <tt>FEN/WATSON</tt><br />
|-<br />
|}<br />
<br />
=== OpenFOAM on BGQ ===<br />
[https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ_OpenFOAM How to use OpenFOAM on BGQ]<br />
<br />
== Python on BlueGene ==<br />
Python 2.7.3 has been installed on BlueGene. To use <span style="color: red;font-weight: bold;">Numpy</span> and <span style="color: red;font-weight: bold;">Scipy</span>, the module <span style="color: red;font-weight: bold;">essl/5.1</span> has to be loaded.<br />
The full python path has to be provided (otherwise the default version is used).<br />
<br />
To use python on BlueGene (from within a job script or a debugjob session):<br />
<source lang="bash"><br />
module load python/2.7.3<br />
##Only if you need numpy/scipy :<br />
module load xlf/14.1 essl/5.1<br />
runjob --np 1 --ranks-per-node=1 --envs HOME=$HOME LD_LIBRARY_PATH=$LD_LIBRARY_PATH PYTHONPATH=/scinet/bgq/tools/Python/python2.7.3-20131205/lib/python2.7/site-packages/ : /scinet/bgq/tools/Python/python2.7.3-20131205/bin/python2.7 /PATHOFYOURSCRIPT.py <br />
</source><br />
<br />
If you want to use the mmap python API, you must use it in PRIVATE mode as shown in the bellow example :<br />
<source lang="bash"><br />
import mmap<br />
mm=mmap.mmap(-1,256,mmap.MAP_PRIVATE)<br />
mm.close()<br />
</source><br />
<br />
Alternatively, you can use the mpi4py and h5py modules.<br />
<br />
Also, please read Cython documentation.<br />
<br />
== Documentation ==<br />
#BGQ Day: Intro to Using the BGQ<br/>[[File:BgqIntro-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html]]<br/>[[Media:BgqintroUpdatedMarch2015.pdf|Slides (updated in 2015) ]] &nbsp;/ &nbsp; [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqintro/bgqintro.mp4 (direct link)]<br />
#BGQ Day: BGQ Hardware Overview<br/>[[File:Bgqhardware-FirstFrame.png|180px|link=http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html]]<br/>[https://support.scinet.utoronto.ca/~northrup/bgqhardware.pdf Slides] &nbsp;/ &nbsp; [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.html Video recording] [http://support.scinet.utoronto.ca/CourseVideo/BGQ/bgqhardware/bgqhardware.mp4 (direct link)]<br />
# [http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JUQUEEN/Documentation/Documention_node.html Julich BGQ Documentation]<br />
# [https://wiki.alcf.anl.gov/parts/index.php/Blue_Gene/Q Argonne Mira BGQ Wiki]<br />
# [https://computing.llnl.gov/tutorials/bgq/ LLNL Sequoia BGQ Info]<br />
# [https://www.alcf.anl.gov/presentations Argonne MiraCon Presentations]<br />
# [http://www.redbooks.ibm.com/redbooks/SG247869/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ System Administration Guide]<br />
# [http://www.redbooks.ibm.com/redbooks/SG247948/wwhelp/wwhimpl/js/html/wwhelp.htm BGQ Application Development ]<br />
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcgetstart.pdf|Getting started]]<br />
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqccompiler.pdf|Compiler reference]]<br />
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqclangref.pdf|Language reference]]<br />
# IBM XL C/C++ for Blue Gene/Q: [[Media:bgqcproguide.pdf|Optimization and Programming Guide]]<br />
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfgetstart.pdf|Getting started]]<br />
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfcompiler.pdf|Compiler reference]]<br />
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqflangref.pdf|Language reference]]<br />
# IBM XL Fortran for Blue Gene/Q: [[Media:bgqfproguide.pdf|Optimization and Programming Guide]]<br />
# [[Media:essl51.pdf|IBM ESSL (Engineering and Scientific Subroutine Library) 5.1 for Linux on Power]]<br />
# [http://content.allinea.com/downloads/userguide.pdf Allinea DDT 4.1 User Guide]<br />
# [https://www.ibm.com/support/knowledgecenter/en/SSFJTW_5.1.0/loadl.v5r1_welcome.html IBM LoadLeveler 5.1]<br />
<br />
<!-- PUT IN TRAC !!!<br />
<br />
=== *Manual Block Creation* ===<br />
<br />
To reconfigure the BGQ nodes you can use the bg_console or the web based navigator from the service node <br />
<br />
<pre><br />
bg_console<br />
</pre><br />
<br />
There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the<br />
following command:<br />
<br />
<pre><br />
gen_small_block <blockid> <midplane> <cnodes> <nodeboard> <br />
gen_small_block R00-M0-N03-32 R00-M0 32 N03<br />
</pre><br />
<br />
The block then needs to be booted using:<br />
<br />
<pre><br />
allocate R00-M0-N03-32<br />
</pre><br />
<br />
If those resources are already booted into another block, that block must be freed before the new block can be <br />
allocated.<br />
<pre><br />
free R00-M0-N03<br />
</pre><br />
<br />
There are many other functions in bg_console:<br />
<br />
<pre><br />
help all<br />
</pre><br />
<br />
The BGQ default nomenclature for hardware is as follows:<br />
<br />
<pre><br />
(R)ack - (M)idplane - (N)ode board or block - (J)node - (C)ore<br />
</pre><br />
<br />
So R00-M01-N03-J00-C02 would correspond to the first rack, second midplane, 3rd block, 1st node, and second core.<br />
<br />
<br />
--!></div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9232BGQ OpenFOAM2018-04-21T02:47:30Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can covert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9231BGQ OpenFOAM2018-04-21T02:46:06Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can covert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9230BGQ OpenFOAM2018-04-21T02:44:35Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can covert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9229BGQ OpenFOAM2018-04-21T02:43:31Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can covert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9228BGQ OpenFOAM2018-04-20T04:09:43Z<p>Fertinaz: /* General Tips and Tricks */</p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing module avail on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"<br />
* Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* You can covert cases from ascii to binary using foamFormatConvert command.<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.<br />
* Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9227BGQ OpenFOAM2018-04-20T03:56:20Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing module avail on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from the "case/constant/polyMesh" not from the "case/processor*/constant/polyMesh"<br />
* Always perform a test run using debug nodes: Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep it under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9226BGQ OpenFOAM2018-04-20T03:54:44Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing module avail on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines the specified regions, snaps out the solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from the "case/constant/polyMesh" not from the "case/processor*/constant/polyMesh"<br />
* Always perform a test run using debug nodes: Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep it under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9225BGQ OpenFOAM2018-04-20T03:51:53Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing module avail on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is hierarchical. If you use this method, then similarly you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines the specified regions, snaps out the solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
This will be updated as the ParaView installation on Niagara will be completed!<br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from the "case/constant/polyMesh" not from the "case/processor*/constant/polyMesh"<br />
* Always perform a test run using debug nodes: Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep it under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.<br />
* OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.<br />
* Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9224BGQ OpenFOAM2018-04-20T03:42:00Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing module avail on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is hierarchical. If you use this method, then similarly you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines the specified regions, snaps out the solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview<br />
<br />
== General Tips and Tricks ==<br />
<br />
* Run serial tasks on FEN using FEN/OpenFOAM/* modules<br />
* Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from the "case/constant/polyMesh" not from the "case/processor*/constant/polyMesh"<br />
* Always perform a test run using debug nodes: Request debug session with "debugjob -i" and use runjob.<br />
* Always work with binary files. This can be set in the "case/system/controlDict".<br />
* Keep your simulations under $SCRATCH.<br />
* If you write your own code, keep it under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.<br />
* If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9223BGQ OpenFOAM2018-04-20T03:30:58Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing module avail on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* blockMesh: Creates the block structured computational volume consists of hex elements.<br />
* decomposePar: Parallelises a serial case. Grid partitioning.<br />
* reconstructPar: Reconstructs a parallel case (results). <br />
* reconstructParMesh: Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is hierarchical. If you use this method, then similarly you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines the specified regions, snaps out the solid areas from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
<br />
# Disabling the pt2pt small message optimizations <br />
export PAMID_SHORT=0<br />
<br />
# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB<br />
export PAMID_EAGER=50M<br />
<br />
# Do not optimise collective comm.<br />
export PAMID_COLLECTIVES=0<br />
<br />
# Do not generate core dump files<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9222BGQ OpenFOAM2018-04-20T03:13:01Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing ''module avail'' on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as ''blockMesh'', ''decomposePar'', ''reconstructParMesh'' you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* ''blockMesh'': Creates the block structured computational volume consists of hex elements.<br />
* ''decomposePar'': Parallelises a serial case. Grid partitioning.<br />
* ''reconstructPar'': Reconstructs a parallel case (results). <br />
* ''reconstructParMesh'': Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called ''decomposeParDict'' in the ''system'' directory of your case folder. ''decomposeParDict'' is the input file for the command ''decomposePar -force''. Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is ''hierarchical''. If you use this method, then similarly you have to define ''hierarchicalCoeffs''. Only difference between ''simple'' and ''hierarchical'' is that with ''hierarchical'' you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.<br />
<br />
The crucial part of the ''decomposeParDict'' is the ''numberOfSubdomains'' defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then ''numberOfSubdomains'' should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.<br />
<br />
== Running Parallel Meshing ==<br />
The built-in meshing tool comes with OpenFOAM package is called ''snappyHexMesh''. This tool reads inputs from the ''system/snappyHexMeshDict'' file and writes outputs to the ''constant/polyMesh'' folder (if used with ''-overwrite'' flag, otherwise writes to separate time folders 1/, 2/). ''snappyHexMesh'' reads the output of ''blockMesh'', refines the specified regions, snaps out the solid regions from the volume and adds boundary layers if enabled. <br />
<br />
Before running mesh generation one needs to run ''decomposePar -force'', so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:<br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = motorBike_mesh<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(jobid).err<br />
# @ output = $(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 06:00:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
# Load modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0<br />
source $FOAM_DOT_FILE<br />
<br />
# Set environment variables<br />
export -n LS_COLORS<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
export PAMID_COLLECTIVES=0<br />
export BG_COREDUMPDISABLED=1<br />
<br />
# Run mesh generation<br />
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel<br />
</pre><br />
<br />
== Loadleveler Submission Script for Solvers ==<br />
<br />
The following is a sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9221BGQ OpenFOAM2018-04-20T02:49:25Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing ''module avail'' on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as ''blockMesh'', ''decomposePar'', ''reconstructParMesh'' you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:<br />
* ''blockMesh'': Creates the block structured computational volume consists of hex elements.<br />
* ''decomposePar'': Parallelises a serial case. Grid partitioning.<br />
* ''reconstructPar'': Reconstructs a parallel case (results). <br />
* ''reconstructParMesh'': Reconstructs a parallel case (mesh). <br />
<br />
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run ''OpenFOAM'' in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. ''OpenFOAM'' has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a ''OpenFOAM'' dictionary called ''decomposeParDict'' in the ''system'' directory of your case folder. ''decomposeParDict'' is the input file for the command ''decomposePar''. Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is ''hierarchical''. If you use this method, then similarly you have to define ''hierarchicalCoeffs''. Only difference between ''simple'' and ''hierarchical'' is that with ''hierarchical'' you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by ''OpenFOAM'' but since this a serial tasks that needs to be performed on ''FEN'', these two methods are suggested.<br />
<br />
The crucial part of the ''decomposeParDict'' is the ''numberOfSubdomains'' defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then ''numberOfSubdomains'' should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise ''OpenFOAM'' will complain because of the mismatch.<br />
<br />
== Loadleveler Submission Script ==<br />
<br />
The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9220BGQ OpenFOAM2018-04-20T02:41:05Z<p>Fertinaz: /* Running Serial OpenFOAM Tasks */</p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing ''module avail'' on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as ''blockMesh'', ''decomposePar'', ''reconstructParMesh'' you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
As it has been written above, if you want to run<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run ''OpenFOAM'' in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. ''OpenFOAM'' has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a ''OpenFOAM'' dictionary called ''decomposeParDict'' in the ''system'' directory of your case folder. ''decomposeParDict'' is the input file for the command ''decomposePar''. Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is ''hierarchical''. If you use this method, then similarly you have to define ''hierarchicalCoeffs''. Only difference between ''simple'' and ''hierarchical'' is that with ''hierarchical'' you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by ''OpenFOAM'' but since this a serial tasks that needs to be performed on ''FEN'', these two methods are suggested.<br />
<br />
The crucial part of the ''decomposeParDict'' is the ''numberOfSubdomains'' defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then ''numberOfSubdomains'' should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise ''OpenFOAM'' will complain because of the mismatch.<br />
<br />
== Loadleveler Submission Script ==<br />
<br />
The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9219BGQ OpenFOAM2018-04-20T02:40:34Z<p>Fertinaz: /* Using OpenFOAM on BG/Q */</p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various OpenFOAM versions installed on BGQ. You can see the list by typing ''module avail'' on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as ''blockMesh'', ''decomposePar'', ''reconstructParMesh'' you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run ''OpenFOAM'' in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. ''OpenFOAM'' has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a ''OpenFOAM'' dictionary called ''decomposeParDict'' in the ''system'' directory of your case folder. ''decomposeParDict'' is the input file for the command ''decomposePar''. Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is ''hierarchical''. If you use this method, then similarly you have to define ''hierarchicalCoeffs''. Only difference between ''simple'' and ''hierarchical'' is that with ''hierarchical'' you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by ''OpenFOAM'' but since this a serial tasks that needs to be performed on ''FEN'', these two methods are suggested.<br />
<br />
The crucial part of the ''decomposeParDict'' is the ''numberOfSubdomains'' defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then ''numberOfSubdomains'' should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise ''OpenFOAM'' will complain because of the mismatch.<br />
<br />
== Loadleveler Submission Script ==<br />
<br />
The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9218BGQ OpenFOAM2018-04-20T00:01:15Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various ''OpenFOAM'' versions installed on BGQ. You can see the list by typing ''module avail'' on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with ''FEN'' refers to the installation on the Front-End-Nodes. Therefore if you want to run serial tasks, e.g ''blockMesh'', ''decomposePar'', ''reconstructParMesh'' you can use ''FEN/OpenFOAM/*'' modules. Please do not forget that ''FEN'' is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you want to decompose a case with 100 million cells, you can occupy the whole ''FEN'' node and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the ''FEN'' using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
== Running Serial OpenFOAM Tasks ==<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run ''OpenFOAM'' in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. ''OpenFOAM'' has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a ''OpenFOAM'' dictionary called ''decomposeParDict'' in the ''system'' directory of your case folder. ''decomposeParDict'' is the input file for the command ''decomposePar''. Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is ''hierarchical''. If you use this method, then similarly you have to define ''hierarchicalCoeffs''. Only difference between ''simple'' and ''hierarchical'' is that with ''hierarchical'' you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by ''OpenFOAM'' but since this a serial tasks that needs to be performed on ''FEN'', these two methods are suggested.<br />
<br />
The crucial part of the ''decomposeParDict'' is the ''numberOfSubdomains'' defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then ''numberOfSubdomains'' should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise ''OpenFOAM'' will complain because of the mismatch.<br />
<br />
== Loadleveler Submission Script ==<br />
<br />
The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9217BGQ OpenFOAM2018-04-19T23:54:49Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various ''OpenFOAM'' versions installed on BGQ. You can see the list by typing ''module avail'' on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with ''FEN'' refers to the installation on the Front-End-Nodes. Therefore if you want to run serial tasks, e.g ''blockMesh'', ''decomposePar'', ''reconstructParMesh'' you can use ''FEN/OpenFOAM/*'' modules. Please do not forget that ''FEN'' is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you want to decompose a case with 100 million cells, you can occupy the whole ''FEN'' node and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the ''FEN'' using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
<br />
== Parallelizing OpenFOAM Cases ==<br />
<br />
In order to run ''OpenFOAM'' in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. ''OpenFOAM'' has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a ''OpenFOAM'' dictionary called ''decomposeParDict'' in the ''system'' directory of your case folder. ''decomposeParDict'' is the input file for the command ''decomposePar''. Below is an example file for decomposing an OpenFOAM case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is ''hierarchical''. If you use this method, then similarly you have to define ''hierarchicalCoeffs''. Only difference between ''simple'' and ''hierarchical'' is that with ''hierarchical'' you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by ''OpenFOAM'' but since this a serial tasks that needs to be performed on ''FEN'', these two methods are suggested.<br />
<br />
The crucial part of the ''decomposeParDict'' is the ''numberOfSubdomains'' defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then ''numberOfSubdomains'' should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise ''OpenFOAM'' will complain because of the mismatch.<br />
<br />
== Loadleveler Submission Script ==<br />
<br />
The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#----------------- Preprocessing on FEN ------------<br />
module purge<br />
module load FEN/OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
$FOAM_APPBIN/blockMesh<br />
$FOAM_APPBIN/decomposePar -force<br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
#----------------- PostProcessing on FEN ------------<br />
module purge<br />
module load FEN/OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
$FOAM_APPBIN/reconstructPar<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9216BGQ OpenFOAM2018-04-19T23:48:41Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various ''OpenFOAM'' versions installed on BGQ. You can see the list by typing ''module avail'' on the terminal:<br />
* OpenFOAM/2.3.1(default)<br />
* OpenFOAM/2.4.0<br />
* OpenFOAM/3.0.1<br />
* OpenFOAM/5.0<br />
and<br />
* FEN/OpenFOAM/2.2.0<br />
* FEN/OpenFOAM/2.3.0<br />
* FEN/OpenFOAM/2.4.0<br />
* FEN/OpenFOAM/3.0.1<br />
* FEN/OpenFOAM/5.0 <br />
<br />
The modules start with ''FEN'' refers to the installation on the Front-End-Nodes. Therefore if you want to run serial tasks, e.g ''blockMesh'', ''decomposePar'', ''reconstructParMesh'' you can use ''FEN/OpenFOAM/*'' modules. Please do not forget that ''FEN'' is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you want to decompose a case with 100 million cells, you can occupy the whole ''FEN'' node and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the ''FEN'' using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
<br />
== Parallel OpenFOAM ==<br />
<br />
In order to run OpenFOAM in parallel the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM 'dictionary' called '''decomposeParDict''' in the system directory of your case folder. Below is an example file for decomposing the OpenFOAM incompressible cavity case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is ''hierarchical''. If you use this method, then similarly you have to define ''hierarchicalCoeffs''. Only difference between ''simple'' and ''hierarchical'' is that with ''hierarchical'' you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by ''OpenFOAM'' but since this a serial tasks that needs to be performed on ''FEN'', these two methods are suggested.<br />
<br />
== Loadleveler Submission Script ==<br />
<br />
The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#----------------- Preprocessing on FEN ------------<br />
module purge<br />
module load FEN/OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
$FOAM_APPBIN/blockMesh<br />
$FOAM_APPBIN/decomposePar -force<br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
#----------------- PostProcessing on FEN ------------<br />
module purge<br />
module load FEN/OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
$FOAM_APPBIN/reconstructPar<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9215BGQ OpenFOAM2018-04-19T23:38:12Z<p>Fertinaz: </p>
<hr />
<div>== Using OpenFOAM on BG/Q ==<br />
There are various ''OpenFOAM'' versions installed on BGQ. You can see the list by typing ''module avail'' on the terminal:<br />
OpenFOAM/2.3.1(default)<br />
OpenFOAM/2.4.0<br />
OpenFOAM/3.0.1<br />
OpenFOAM/5.0<br />
<br />
FEN/OpenFOAM/2.2.0<br />
FEN/OpenFOAM/2.3.0<br />
FEN/OpenFOAM/2.4.0<br />
FEN/OpenFOAM/3.0.1<br />
FEN/OpenFOAM/5.0 <br />
<br />
The modules start with ''FEN'' refers to the installation on the Front-End-Nodes. Therefore if you want to run serial tasks, e.g ''blockMesh'', ''decomposePar'', ''reconstructParMesh'' you can use ''FEN/OpenFOAM/*'' modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you want to decompose a case with 100 million cells, you can occupy the whole login/FEN node and probably run out of memory.<br />
<br />
When you want to submit a job, you should do that on the ''FEN'' using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.<br />
<br />
<br />
== Parallel OpenFOAM ==<br />
<br />
In order to run OpenFOAM in parallel the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that performs this operation. The control for this is done creating a OpenFOAM 'dictionary' called '''decomposeParDict''' in the system directory of your case folder. Below is an example file for decomposing the OpenFOAM incompressible cavity case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
Another option for decomposition method is ''hierarchical''. If you use this method, then similarly you have to define ''hierarchicalCoeffs''. Only difference between ''simple'' and ''hierarchical'' is that with ''hierarchical'' you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by ''OpenFOAM'' but since this a serial tasks that needs to be performed on ''FEN'', these two methods are suggested.<br />
<br />
== Loadleveler Submission Script ==<br />
<br />
The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#----------------- Preprocessing on FEN ------------<br />
module purge<br />
module load FEN/OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
$FOAM_APPBIN/blockMesh<br />
$FOAM_APPBIN/decomposePar -force<br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
#----------------- PostProcessing on FEN ------------<br />
module purge<br />
module load FEN/OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
$FOAM_APPBIN/reconstructPar<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=BGQ_OpenFOAM&diff=9178BGQ OpenFOAM2018-03-01T18:13:59Z<p>Fertinaz: </p>
<hr />
<div><br />
== Parallel OpenFOAM ==<br />
<br />
In order to run OpenFOAM in parallel the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a '''[http://www.openfoam.org/docs/user/running-applications-parallel.php decomposePar]''' utility that will perform this operation after intial mesh creation. The control for this is done creating a OpenFOAM 'dictionary' called '''decomposeParDict''' in the system directory of your case folder. Below is an example file for decomposing the OpenFOAM incompressible cavity case for running on 4 cores.<br />
<br />
'''system/decomposeParDict'''<br />
<pre><br />
<br />
/*--------------------------------*- C++ -*----------------------------------*\<br />
| ========= | |<br />
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |<br />
| \\ / O peration | Version: 2.4.0 |<br />
| \\ / A nd | Web: www.OpenFOAM.org |<br />
| \\/ M anipulation | |<br />
\*---------------------------------------------------------------------------*/<br />
FoamFile<br />
{<br />
version 2.0;<br />
format ascii;<br />
class dictionary;<br />
location "system";<br />
object decomposeParDict;<br />
}<br />
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //<br />
<br />
numberOfSubdomains 4;<br />
<br />
method simple;<br />
<br />
simpleCoeffs<br />
{<br />
n ( 2 2 1 );<br />
delta 0.001;<br />
}<br />
<br />
distributed no;<br />
<br />
roots ( );<br />
<br />
<br />
// ************************************************************************* //<br />
<br />
</pre><br />
<br />
== Loadleveler Submission Script ==<br />
<br />
The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the<br />
pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob,<br />
and then reconstructing the solution again using the front-end node. <br />
<br />
<pre><br />
#!/bin/sh<br />
# @ job_name = bgqopenfoam<br />
# @ job_type = bluegene<br />
# @ comment = "BGQ Job By Size"<br />
# @ error = $(job_name).$(Host).$(jobid).err<br />
# @ output = $(job_name).$(Host).$(jobid).out<br />
# @ bg_size = 64<br />
# @ wall_clock_limit = 30:00<br />
# @ bg_connectivity = Torus<br />
# @ queue <br />
<br />
#----------------- Preprocessing on FEN ------------<br />
module purge<br />
module load FEN/OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
$FOAM_APPBIN/blockMesh<br />
$FOAM_APPBIN/decomposePar -force<br />
<br />
#------------------ Solver on BGQ --------------------<br />
# Load BGQ OpenFOAM modules<br />
module purge<br />
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob<br />
# so removing LS_COLORS should free up enough space<br />
export -n LS_COLORS<br />
# Some solvers, simpleFOAM particularly, will hang on startup when using the default<br />
# network parameters. Disabling the pt2pt small message optimizations seems to allow it to run.<br />
export PAMID_SHORT=0<br />
export PAMID_EAGER=50M<br />
<br />
# Run solver<br />
runjob --np 4 --env-all : $FOAM_APPBIN/icoFoam -parallel<br />
<br />
#----------------- PostProcessing on FEN ------------<br />
module purge<br />
module load FEN/OpenFOAM/2.4.0<br />
source $FOAM_DOT_FILE<br />
<br />
$FOAM_APPBIN/reconstructPar<br />
<br />
</pre><br />
<br />
== Post-Processing== <br />
<br />
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=SOSCIP_GPU&diff=9137SOSCIP GPU2018-02-01T21:52:18Z<p>Fertinaz: </p>
<hr />
<div>{{Infobox Computer<br />
|image=[[Image:S882lc.png|center|300px|thumb]]<br />
|name=SOSCIP GPU <br />
|installed=September 2017<br />
|operatingsystem= Ubuntu 16.04 le <br />
|loginnode= sgc01 <br />
|nnodes= 14x Power 8 with 4x NVIDIA P100<br />
|rampernode=512 GB<br />
|corespernode= 2 x 10core (20 physical, 160 SMT)<br />
|interconnect=Infiniband EDR <br />
|vendorcompilers=xlc/xlf, nvcc<br />
}}<br />
<br />
== SOSCIP ==<br />
<br />
The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].<br />
<br />
== Support Email ==<br />
<br />
Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.<br />
<br />
== Specifications==<br />
<br />
The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.<br />
<br />
== Access and Login ==<br />
<br />
In order to obtain access to the system, you must request access to the SOSCIP GPU Platform. Instructions will have been sent to your sponsoring faculty member via E-mail at the beginning of your SOSCIP project.<br />
<br />
Access to the SOSCIP GPU Platform is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>''' via ssh. Your user name and password is the same as it is for SciNet systems.<br />
<br />
== Filesystem ==<br />
<br />
The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.<br />
<br />
== Job Submission ==<br />
<br />
The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.<br />
<br />
<pre><br />
$ sbatch myjob.script<br />
</pre><br />
<br />
Where myjob.script is <br />
<br />
<pre><br />
#!/bin/bash<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
cd $SLURM_SUBMIT_DIR<br />
<br />
hostname<br />
nvidia-smi<br />
</pre><br />
<br />
You can queury job information using<br />
<br />
<pre><br />
squeue<br />
</pre><br />
<br />
To cancel a job use<br />
<br />
<pre><br />
scancel $JOBID<br />
</pre><br />
<br />
=== Longer jobs ===<br />
<br />
If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.<br />
<br />
=== Interactive ===<br />
<br />
For an interactive session use<br />
<br />
<pre><br />
salloc --gres=gpu:4<br />
</pre><br />
<br />
=== Automatic Re-submission and Job Dependencies ===<br />
<br />
Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation. <br />
<br />
<pre><br />
#!/bin/bash <br />
<br />
#SBATCH --nodes=1 <br />
#SBATCH --ntasks=20 # MPI tasks (needed for srun) <br />
#SBATCH --time=00:10:00 # H:M:S<br />
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node<br />
<br />
cd $SLURM_SUBMIT_DIR<br />
<br />
: ${job_number:="1"} # set job_nubmer to 1 if it is undefined<br />
job_number_max=3<br />
<br />
echo "hi from ${SLURM_JOB_ID}"<br />
<br />
#RUN JOB HERE<br />
<br />
<br />
# SUBMIT NEXT JOB<br />
if [[ ${job_number} -lt ${job_number_max} ]]<br />
then<br />
(( job_number++ ))<br />
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")<br />
echo "submitted ${next_jobid}"<br />
fi<br />
<br />
sleep 15<br />
<br />
echo "${SLURM_JOB_ID} done"<br />
<br />
</pre><br />
<br />
== Software Installed ==<br />
<br />
=== IBM PowerAI ===<br />
<br />
The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.<br />
<br />
==== GNU Compilers ====<br />
<br />
More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:<br />
<br />
Advanced Toolchain V10.0<br />
<pre><br />
module load gcc/6.3.1<br />
</pre><br />
<br />
Advanced Toolchain V11.0<br />
<pre><br />
module load gcc/7.2.1<br />
</pre><br />
<br />
More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]<br />
<br />
==== IBM XL Compilers ====<br />
<br />
To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run<br />
<br />
<pre><br />
module load xlc/13.1.5<br />
module load xlf/15.1.5<br />
</pre><br />
<br />
IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.<br />
<br />
Information about the IBM XL Compilers can be found at the following links:<br />
<br />
[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]<br />
<br />
[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]<br />
<br />
==== NVIDIA GPU Driver Version ====<br />
<br />
The current NVIDIA driver version is 384.66<br />
<br />
==== CUDA ====<br />
<br />
The current installed CUDA Tookits is are version 8.0 and version 9.0<br />
<br />
<pre><br />
module load cuda/8.0<br />
</pre><br />
<br />
or <br />
<br />
<pre><br />
module load cuda/9.0<br />
</pre><br />
<br />
<br />
The CUDA driver is installed locally, however the CUDA Toolkit is installed in:<br />
<br />
<pre><br />
/usr/local/cuda-8.0<br />
/usr/local/cuda-9.0<br />
</pre><br />
<br />
Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.<br />
<br />
Documentation and API reference information for the CUDA Toolkit can be found here: [http://docs.nvidia.com/cuda/index.html http://docs.nvidia.com/cuda/index.html]<br />
<br />
==== OpenMPI ====<br />
<br />
Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.<br />
<br />
<pre><br />
$ module load openmpi/2.1.1-gcc-5.4.0<br />
$ module load openmpi/2.1.1-XL-13_15.1.5<br />
</pre><br />
<br />
== Other Software ==<br />
<br />
Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).<br />
<br />
In the following subsections are instructions for installing several common software packages.<br />
<br />
=== Anaconda (Python) ===<br />
<br />
Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.<br />
<br />
Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]<br />
<br />
NOTE: Be sure to download the '''Power8''' installer.<br />
<br />
TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda<br />
<br />
=== Keras ===<br />
<br />
Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.<br />
<br />
The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.<br />
<br />
Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.<br />
<br />
=== PyTorch ===<br />
<br />
PyTorch is the Python implementation of the Torch framework for deep learning. <br />
<br />
It is suggested that you use PyTorch within Anaconda.<br />
<br />
There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.<br />
<br />
Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]<br />
<br />
Before building PyTorch, make sure to load cuda by running <br />
<br />
<pre><br />
module load cuda/8.0<br />
</pre><br />
<br />
NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.<br />
<br />
== LINKS ==<br />
<br />
[https://www.olcf.ornl.gov/kb_articles/summitdev-quickstart/#System_Overview Summit Dev System at ORNL]<br />
<br />
== DOCUMENTATION ==<br />
<br />
# GPU Cluster Introduction: [[Media:SOSCIP_GPU_Platform.pdf|SOSCIP GPU Platform]]</div>Fertinazhttps://oldwiki.scinet.utoronto.ca/index.php?title=File:SOSCIP_GPU_Platform.pdf&diff=9136File:SOSCIP GPU Platform.pdf2018-02-01T21:51:43Z<p>Fertinaz: </p>
<hr />
<div></div>Fertinaz