BGQ OpenFOAM

Using OpenFOAM on BG/Q

There are various OpenFOAM versions installed on BGQ. You can see the list by typing module avail on the terminal:

• OpenFOAM/2.3.1(default)
• OpenFOAM/2.4.0
• OpenFOAM/3.0.1
• OpenFOAM/5.0

and

• FEN/OpenFOAM/2.2.0
• FEN/OpenFOAM/2.3.0
• FEN/OpenFOAM/2.4.0
• FEN/OpenFOAM/3.0.1
• FEN/OpenFOAM/5.0

The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.

When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.

Running Serial OpenFOAM Tasks

As it has been written in the previous section, if you want to run serial tasks you need to use one of the FEN based modules. Most common serial tasks are:

• blockMesh: Creates the block structured computational volume consists of hex elements.
• decomposePar: Parallelises a serial case. Grid partitioning.
• reconstructPar: Reconstructs a parallel case (results).
• reconstructParMesh: Reconstructs a parallel case (mesh).

These binaries are not available on the compute nodes, therefore you can use these tools only on the FEN anyway.

Parallelizing OpenFOAM Cases

In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a decomposePar utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command "decomposePar -force". Below is an example file for decomposing an OpenFOAM case for running on 4 cores.

system/decomposeParDict

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.4.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 4;

method          simple;

simpleCoeffs
{
    n               ( 2 2 1 );
    delta           0.001;
}

// ************************************************************************* //

Another option for decomposition method is hierarchical. If you use this method, then similarly you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical method you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.

The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.

Running Parallel Meshing

The built-in meshing tool comes with OpenFOAM package is called snappyHexMesh. This tool reads inputs from the "system/snappyHexMeshDict" file and writes outputs to the "constant/polyMesh" folder (if used with -overwrite flag, otherwise writes to separate time folders 1/, 2/). snappyHexMesh operates on the outputs of blockMesh, refines the specified regions, snaps out the solid areas from the volume and adds boundary layers if enabled.

Before running mesh generation one needs to run "decomposePar -force", so that the case is parallelised and made available to run parallel executions on it. One can submit the script below to run parallel mesh generation on BG/Q:

#!/bin/sh
# @ job_name           = motorBike_mesh
# @ job_type           = bluegene
# @ comment            = "BGQ Job By Size"
# @ error              = $(jobid).err
# @ output             = $(jobid).out
# @ bg_size            = 64
# @ wall_clock_limit   = 06:00:00
# @ bg_connectivity    = Torus
# @ queue 

# Load modules
module purge
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0
source $FOAM_DOT_FILE

# NOTE: when using --env-all there is a limit of 8192 characters that can be passed to runjob
# so removing LS_COLORS should free up enough space
export -n LS_COLORS

# Disabling the pt2pt small message optimizations  
export PAMID_SHORT=0

# Sets the cutoff point for switching from eager to rendezvous protocol at 50MB
export PAMID_EAGER=50M

# Do not optimise collective comm.
export PAMID_COLLECTIVES=0

# Do not generate core dump files
export BG_COREDUMPDISABLED=1

# Run mesh generation
runjob --np 1024 --ranks-per-node=16 --env-all : $FOAM_APPBIN/snappyHexMesh -overwrite -parallel

Loadleveler Submission Script for Solvers

The following is a sample script for running the OpenFOAM tutorial case on BG/Q:

#!/bin/sh
# @ job_name           = bgqopenfoam
# @ job_type           = bluegene
# @ comment            = "BGQ Job By Size"
# @ error              = $(job_name).$(Host).$(jobid).err
# @ output             = $(job_name).$(Host).$(jobid).out
# @ bg_size            = 64
# @ wall_clock_limit   = 06:00:00
# @ bg_connectivity    = Torus
# @ queue 

#------------------ Solver on BGQ --------------------
# Load BGQ OpenFOAM modules
module purge
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/5.0
source $FOAM_DOT_FILE

# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob
# so removing LS_COLORS should free up enough space
export -n LS_COLORS
# Some solvers, simpleFOAM particularly, will hang on startup when using the default
# network parameters.  Disabling the pt2pt small message optimizations seems to allow it to run.
export PAMID_SHORT=0
export PAMID_EAGER=50M

# Run solver
runjob --np 1024 --env-all  : $FOAM_APPBIN/icoFoam -parallel

Post-Processing

https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview