BGQ OpenFOAM

Example OpenFOAM on BGQ

To run OpenFOAM on the BGQ

Sample script for running the OpenFOAM tutorial case on BGQ

Parallel OpenFOAM

In order to run OpenFOAM in parallel the problem needs to be decomposed to to match the number of processor cores to be used. OpenFOAM has a [ http://www.openfoam.org/docs/user/running-applications-parallel.php decompsePar] utility that will perform this operation after intial mesh creation. The control for this is done creating a OpenFOAM 'dictionary' called decomposeParDict in the system directory of your case folder. Below is an example file for decomposing the cavity case for running on 16 cores.

system/decomposeParDict

/*--------------------------------*- C++ -*----------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.1.1                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
FoamFile
{
    version     2.0;
    format      ascii;
    class       dictionary;
    location    "system";
    object      decomposeParDict;
}
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //

numberOfSubdomains 16;

method          simple;

simpleCoeffs
{
    n               ( 4 4 1 );
    delta           0.001;
}

distributed     no;

roots           ( );


// ************************************************************************* //

Loadleveler Submission Script

The following is a Sample script for running the OpenFOAM tutorial case on BGQ, first doing the pre-processing serially on the front-end p7 node, then running the case on the BGQ using runjob, and then reconstructing the solution again using the front-end node.

#!/bin/sh
# @ job_name           = bgqopenfoam
# @ job_type           = bluegene
# @ comment            = "BGQ Job By Size"
# @ error              = $(job_name).$(Host).$(jobid).err
# @ output             = $(job_name).$(Host).$(jobid).out
# @ bg_size            = 64
# @ wall_clock_limit   = 30:00
# @ bg_connectivity    = Torus
# @ queue 

#----------------- Preprocessing on FEN ------------
module purge
module load FEN/OpenFOAM
source $FOAM_DOT_FILE

$FOAM_APPBIN/blockMesh
$FOAM_APPBIN/decomposePar

#------------------ Solver on BGQ --------------------
# Load BGQ OpenFOAM modules
module purge
module load binutils/2.23 bgqgcc/4.8.1 mpich2/gcc-4.8.1 OpenFOAM
source $FOAM_DOT_FILE

# Run solver
runjob --np 16 --ranks-per-node=1 --env-all  : $FOAM_APPBIN/icoFoam -parallel

#----------------- PostProcessing on FEN ------------
module purge
module load FEN/OpenFOAM
source $FOAM_DOT_FILE

$FOAM_APPBIN/reconstructPar

Post-Processing

https://support.scinet.utoronto.ca/wiki/index.php/Using_Paraview