BGQ OpenFOAM
Using OpenFOAM on BG/Q
There are various OpenFOAM versions installed on BGQ. You can see the list by typing "module avail" on the terminal:
- OpenFOAM/2.3.1(default)
- OpenFOAM/2.4.0
- OpenFOAM/3.0.1
- OpenFOAM/5.0
and
- FEN/OpenFOAM/2.2.0
- FEN/OpenFOAM/2.3.0
- FEN/OpenFOAM/2.4.0
- FEN/OpenFOAM/3.0.1
- FEN/OpenFOAM/5.0
The modules start with FEN refer to the installations can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar or reconstructParMesh, please use FEN/OpenFOAM/* modules. Do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and run out of memory therefore make it unavailable for everyone.
When you want to submit a job, you should do that on the FEN using a batch script. This is the only way of using compute nodes on BGQ. There is a sample batch script below. You can use it as a template and modify it according to your needs.
Running Serial OpenFOAM Tasks
As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:
- blockMesh: Creates the block structured computational volume consists of hex elements.
- decomposePar: Parallelises a serial case. Grid partitioning.
- reconstructPar: Reconstructs a parallel case (results).
- reconstructParMesh: Reconstructs a parallel case (mesh).
These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.
Parallelizing OpenFOAM Cases
In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a decomposePar utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.
system/decomposeParDict
/*--------------------------------*- C++ -*----------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: 2.4.0 |
| \\ / A nd | Web: www.OpenFOAM.org |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
FoamFile
{
version 2.0;
format ascii;
class dictionary;
location "system";
object decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
numberOfSubdomains 4;
method simple;
simpleCoeffs
{
n ( 2 2 1 );
delta 0.001;
}
// ************************************************************************* //
Another option for decomposition is hierarchical. If you use this method, then similar to simple you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation (xyz or zyx). There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.
The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.
Running Parallel Meshing
The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines specified regions, snaps out solid areas from the volume and adds boundary layers if enabled.
Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:
#!/bin/sh # @ job_name = motorBike_mesh # @ job_type = bluegene # @ comment = "BGQ Job By Size" # @ error = $(jobid).err # @ output = $(jobid).out # @ bg_size = 64 # @ wall_clock_limit = 06:00:00 # @ bg_connectivity = Torus # @ queue # Load modules module purge module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0 source $FOAM_DOT_FILE # NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob # so removing LS_COLORS should free up enough space export -n LS_COLORS # Disabling the pt2pt small message optimizations export PAMID_SHORT=0 # Sets the cutoff point for switching from eager to rendezvous protocol at 50MB export PAMID_EAGER=50M # Do not optimise collective comm. export PAMID_COLLECTIVES=0 # Do not generate core dump files export BG_COREDUMPDISABLED=1 # Run mesh generation runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel
Loadleveler Submission Script for Solvers
The following is a sample script for running the OpenFOAM tutorial case on BG/Q:
#!/bin/sh # @ job_name = bgqopenfoam # @ job_type = bluegene # @ comment = "BGQ Job By Size" # @ error = $(job_name).$(Host).$(jobid).err # @ output = $(job_name).$(Host).$(jobid).out # @ bg_size = 64 # @ wall_clock_limit = 06:00:00 # @ bg_connectivity = Torus # @ queue #------------------ Solver on BGQ -------------------- # Load BGQ OpenFOAM modules module purge module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0 source $FOAM_DOT_FILE # NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob # so removing LS_COLORS should free up enough space export -n LS_COLORS # Some solvers, simpleFOAM particularly, will hang on startup when using the default # network parameters. Disabling the pt2pt small message optimizations seems to allow it to run. export PAMID_SHORT=0 export PAMID_EAGER=50M # Do not optimise collective comm. export PAMID_COLLECTIVES=0 # Do not generate core dump files export BG_COREDUMPDISABLED=1 # Run solver runjob --np 1024 --env-all : $FOAM_APPBIN/icoFoam -parallel
Post-Processing
This will be updated as the ParaView installation on Niagara will be completed!
https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview
General Tips and Tricks
- Run serial tasks on FEN using FEN/OpenFOAM/* modules
- Make a quality check for your mesh using checkMesh tool. Be careful that if you run a serial checkMesh in a parallel case, it will only return results from "case/constant/polyMesh" not from "case/processor*/constant/polyMesh"
- Perform test runs using debug nodes before you submit large jobs. Request debug session with "debugjob -i" and use runjob.
- Always work with binary files. This can be set in the "case/system/controlDict".
- You can covert cases from ascii to binary using foamFormatConvert command.
- Keep your simulations under $SCRATCH.
- If you write your own code, keep them under $HOME. Preferably create a directory "$HOME/OpenFOAM/username-X.Y/src" and work here.
- If you write your own code, do not forget to compile them to $FOAM_USER_APPBIN or $FOAM_USER_LIBBIN. You might need to compile shared objects on debug nodes as well.
- OpenFOAM is a pure MPI code, there is no multithreading in OpenFOAM.
- Each and every node on BG/Q has 16 GB memory and 16 compute cores. Some OpenFOAM functions, especially snappyHexMesh, are very memory consuming up to 4GB memory per 1M cells. Use 8 ranks per node if you run out of memory however be careful with that. Do not waste resources. Usually solvers require 1GB memory per 1M cells.
- Try collated option using the version 5.0. It significantly reduces the number of files however master processor gets overloaded.