BGQ OpenFOAM

Using OpenFOAM on BG/Q

There are various OpenFOAM versions installed on BGQ. You can see the list by typing module avail on the terminal:

• OpenFOAM/2.3.1(default)
• OpenFOAM/2.4.0
• OpenFOAM/3.0.1
• OpenFOAM/5.0

and

• FEN/OpenFOAM/2.2.0
• FEN/OpenFOAM/2.3.0
• FEN/OpenFOAM/2.4.0
• FEN/OpenFOAM/3.0.1
• FEN/OpenFOAM/5.0

The modules start with FEN refer to the installation that can be used on the Front-End-Nodes. Therefore if you want to run serial tasks such as blockMesh, decomposePar, reconstructParMesh you can use FEN/OpenFOAM/* modules. Please do not forget that FEN is not a dedicated area, each Front-End-Node is shared among connected users and only has 32GB of memory. So if you try to decompose a case with 100 million cells, you will occupy the whole FEN machine and probably run out of memory.

When you want to submit a job, you should do that on the FEN using a batch script. There is a sample batch script below. You can use it as a template and modify it according to your needs.

Running Serial OpenFOAM Tasks

As it has been written above, if you want to run

Parallelizing OpenFOAM Cases

In order to run OpenFOAM in parallel, the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a decomposePar utility that performs this operation. The control for this is done creating a OpenFOAM dictionary called decomposeParDict in the system directory of your case folder. decomposeParDict is the input file for the command decomposePar. Below is an example file for decomposing an OpenFOAM case for running on 4 cores.

system/decomposeParDict

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.4.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 4;

method          simple;

simpleCoeffs
{
    n               ( 2 2 1 );
    delta           0.001;
}

// ************************************************************************* //

Another option for decomposition method is hierarchical. If you use this method, then similarly you have to define hierarchicalCoeffs. Only difference between simple and hierarchical is that with hierarchical you can define the order of the decomposition operation. There are more complicated methods of decomposition supported by OpenFOAM but since this a serial tasks that needs to be performed on FEN, these two methods are suggested.

The crucial part of the decomposeParDict is the numberOfSubdomains defined in the file. The intended number of cores should match this value. Therefore if one wants to run a case on 64 nodes using all cores then numberOfSubdomains should be 1024. Also, multiplication of the n values should be equal to this number for consistency. Otherwise OpenFOAM will complain because of the mismatch.

Loadleveler Submission Script

The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob, and then reconstructing the solution again using the front-end node.

#!/bin/sh
# @ job_name           = bgqopenfoam
# @ job_type           = bluegene
# @ comment            = "BGQ Job By Size"
# @ error              = $(job_name).$(Host).$(jobid).err
# @ output             = $(job_name).$(Host).$(jobid).out
# @ bg_size            = 64
# @ wall_clock_limit   = 30:00
# @ bg_connectivity    = Torus
# @ queue 

#------------------ Solver on BGQ --------------------
# Load BGQ OpenFOAM modules
module purge
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM/2.4.0
source $FOAM_DOT_FILE

# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob
# so removing LS_COLORS should free up enough space
export -n LS_COLORS
# Some solvers, simpleFOAM particularly, will hang on startup when using the default
# network parameters.  Disabling the pt2pt small message optimizations seems to allow it to run.
export PAMID_SHORT=0
export PAMID_EAGER=50M

# Run solver
runjob --np 4 --env-all  : $FOAM_APPBIN/icoFoam -parallel

Post-Processing

https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview