Namd on BGQ

From oldwiki.scinet.utoronto.ca
Jump to navigation Jump to search

A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation (Cytochrome c Oxidase embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms.

Performance Tuning Benchmarks

Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three Benchmark time lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file).

Ranks Cores NAMD Config Options ns/day Efficiency
16 512 2.79 1.00
16 1024 5.05 0.91
16 1024 twoAwayX (default) 5.62 1.01
16 2048 twoAwayX (default) 10.07 0.90
16 2048 twoAwayXY 10.59 0.95
16 4096 twoAwayX 14.32 0.64
16 4096 twoAwayXY (default) 17.63 0.79
16 4096 twoAwayXYZ 16.79 0.75
16 8192 twoAwayX 23.52 0.53
16 8192 twoAwayXY (default) 25.00 0.56
16 16384 twoAwayX 23.67 0.27
16 16384 twoAwayXY 28.31 0.32
16 16384 twoAwayXYZ (default) 27.98 0.31

PME Pencils

A "pencil-based" PME decomposition may be more efficient than the default "slab-based decomposition". In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.

Ranks Cores NAMD Config Options ns/day Efficiency
16 4096 twoAwayXY, PMEPencils=8, lblUnload=yes 12.93 0.58
16 4096 twoAwayXY, PMEPencils=12, lblUnload=yes 17.27 0.77
16 4096 twoAwayXY, PMEPencils=16, lblUnload=yes 16.02 0.72
16 4096 twoAwayXY, PMEPencils=20, lblUnload=yes 15.41 0.69
16 4096 twoAwayXY, PMEPencils=12 16.21 0.73
16 4096 twoAwayXY, PMEPencils=16 17.92 0.80
16 4096 twoAwayXY, PMEPencils=20 17.99 0.81
16 4096 twoAwayXY, PMEPencils=24 17.83 0.80
16 4096 twoAwayXY, PMEPencils=36 16.97 0.76
8 4096 twoAwayXY, PMEPencils=20 18.24 0.82
16 4096 twoAwayXY, PMEPencils=20 17.99 0.81
32 4096 twoAwayXY, PMEPencils=20 13.94 0.63

Ranks-Per-Node Study

The "ranks-per-node" or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag "ldbUnloadZero yes" which supposedly ensures no work is scheduled for processor zero.

Ranks Cores NAMD Config Options ns/day Efficiency
4 512 2.86 1.03
8 512 2.84 1.02
16 512 2.79 1.00
32 512 2.29 0.82
16 512 ldbUnloadZero=yes 2.79 1.00

Incorrect Particle-Mesh Ewald Grid

Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the pmeGridSpacing 1.0 setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.

Ranks Cores NAMD Config Options ns/day Efficiency
16 512 Poor PME Multiple (144x144x111) 2.70 0.97
16 1024 Poor PME Multiple (144x144x111) 5.13 0.92
16 2048 Poor PME Multiple (144x144x111) 8.61 0.77
16 4096 Poor PME Multiple (144x144x111) 13.93 0.62
16 8192 Poor PME Multiple (144x144x111) 17.08 0.38
16 16384 Poor PME Multiple (144x144x111) 17.64 0.20

Documentation

  1. NAMD 2.9 User Guide
  2. NAMD Performance Tuning Wiki