BGQ OpenFOAM

Parallel OpenFOAM

In order to run OpenFOAM in parallel the problem needs to be decomposed into a number of subdomains that match the number of processors that will be used. OpenFOAM has a decompsePar utility that will perform this operation after intial mesh creation. The control for this is done creating a OpenFOAM 'dictionary' called decomposeParDict in the system directory of your case folder. Below is an example file for decomposing the OpenFOAM incompressible cavity case for running on 4 cores.

system/decomposeParDict

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.2.0                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 4;

method          simple;

simpleCoeffs
{
    n               ( 2 2 1 );
    delta           0.001;
}

distributed     no;

roots           ( );


// ************************************************************************* //

Loadleveler Submission Script

The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob, and then reconstructing the solution again using the front-end node.

#!/bin/sh
# @ job_name           = bgqopenfoam
# @ job_type           = bluegene
# @ comment            = "BGQ Job By Size"
# @ error              = $(job_name).$(Host).$(jobid).err
# @ output             = $(job_name).$(Host).$(jobid).out
# @ bg_size            = 64
# @ wall_clock_limit   = 30:00
# @ bg_connectivity    = Torus
# @ queue 

#----------------- Preprocessing on FEN ------------
module purge
module load FEN/OpenFOAM
source $FOAM_DOT_FILE

$FOAM_APPBIN/blockMesh
$FOAM_APPBIN/decomposePar -force

#------------------ Solver on BGQ --------------------
# Load BGQ OpenFOAM modules
module purge
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM
source $FOAM_DOT_FILE

# NOTE: when using --env-all there is a limit of 8192 characters that can passed to runjob
# so removing LS_COLORS should free up enough space
export -n LS_COLORS
# Some solvers, simpleFOAM particularly, will hang on startup when using the default
# network parameters.  Disabling the pt2pt small message optimizations seems to allow it to run.
export PAMID_SHORT=0
export PAMID_EAGER=50M

# Run solver
runjob --np 4 --env-all  : $FOAM_APPBIN/icoFoam -parallel

#----------------- PostProcessing on FEN ------------
module purge
module load FEN/OpenFOAM
source $FOAM_DOT_FILE

$FOAM_APPBIN/reconstructPar

Post-Processing

https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview