https://oldwiki.scinet.utoronto.ca/api.php?action=feedcontributions&feedformat=atom&user=Cing 2026-07-10T07:47:21Z User contributions MediaWiki 1.35.12 https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5474 2012-11-20T18:49:33Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (32, 64, 128, etc.). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. The column &quot;cores&quot; in the following results indicates the number of &quot;physical cores&quot; specified by the '''bg_size''' parameter on the submission script.<br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 32 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |32<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |64<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |64<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |128<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |128<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |256<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |256<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |512<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |512<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |1024<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |1024<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |256<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |256<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |32<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |64<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |128<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |256<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |32<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |64<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |128<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |256<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> Comparing 32/ppn4 with 64/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 128/ppn4 simulations showed poor scaling so larger systems were not tested.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |32<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |32<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |32<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |32<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |32<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |32<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |32<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |32<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |64<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |64<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |64<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |64<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |64<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |128<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |128<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |128<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |128<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> In an attempt to maximize the number of processes per node (64), one may also utilize less ranks-per-node, such as 2,4 or 8, while requesting a larger number of threads. The performance of using 8-ranks per node on 64-physical nodes with ppn 8 is approximately equal to the performance of the 16-ranks-per-node above. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |8<br /> |32<br /> |ppn 1<br /> |1.53<br /> |0.56<br /> |----<br /> |8<br /> |32<br /> |ppn 2<br /> |2.77<br /> |1.01<br /> |----<br /> |8<br /> |32<br /> |ppn 4<br /> |4.64<br /> |1.69<br /> |----<br /> |8<br /> |32<br /> |ppn 8<br /> |6.32<br /> |2.30<br /> |----<br /> |8<br /> |64<br /> |ppn 1<br /> |2.79<br /> |0.51<br /> |----<br /> |8<br /> |64<br /> |ppn 2<br /> |5.59<br /> |1.02<br /> |----<br /> |8<br /> |64<br /> |ppn 4<br /> |8.25<br /> |1.50<br /> |----<br /> |8<br /> |64<br /> |ppn 8<br /> |10.71<br /> |1.95<br /> |----<br /> <br /> |}<br /> <br /> Simulations of both 2 and 4 ranks-per-node offered comparable simulation efficiencies to the other full 64 process jobs in this section. The combination of 32 ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. A simulations of ppn=64 with 1 rank-per-node was not successful, and the ppn=32 job required the use of PMEPencils.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |2<br /> |32<br /> |ppn 32<br /> |6.35<br /> |2.29<br /> |----<br /> |2<br /> |32<br /> |ppn 32 (only with PMEPencils 8)<br /> |7.80<br /> |1.95<br /> |----<br /> |4<br /> |32<br /> |ppn 16<br /> |6.30<br /> |2.31<br /> |----<br /> |4<br /> |32<br /> |ppn 16<br /> |10.73<br /> |1.42<br /> |----<br /> |32<br /> |32<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |32<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |32<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |32<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |32<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |32<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5473 2012-11-20T18:47:53Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (32, 64, 128, etc.). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 32 physical core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |32<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |64<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |64<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |128<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |128<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |256<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |256<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |512<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |512<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |1024<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |1024<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |256<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |256<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |256<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |32<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |64<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |128<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |256<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |32<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |64<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |128<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |256<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> Comparing 32/ppn4 with 64/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 128/ppn4 simulations showed poor scaling so larger systems were not tested.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |32<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |32<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |32<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |32<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |32<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |32<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |32<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |32<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |64<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |64<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |64<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |64<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |64<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |128<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |128<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |128<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |128<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> In an attempt to maximize the number of processes per node (64), one may also utilize less ranks-per-node, such as 2,4 or 8, while requesting a larger number of threads. The performance of using 8-ranks per node on 64-physical nodes with ppn 8 is approximately equal to the performance of the 16-ranks-per-node above. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |8<br /> |32<br /> |ppn 1<br /> |1.53<br /> |0.56<br /> |----<br /> |8<br /> |32<br /> |ppn 2<br /> |2.77<br /> |1.01<br /> |----<br /> |8<br /> |32<br /> |ppn 4<br /> |4.64<br /> |1.69<br /> |----<br /> |8<br /> |32<br /> |ppn 8<br /> |6.32<br /> |2.30<br /> |----<br /> |8<br /> |64<br /> |ppn 1<br /> |2.79<br /> |0.51<br /> |----<br /> |8<br /> |64<br /> |ppn 2<br /> |5.59<br /> |1.02<br /> |----<br /> |8<br /> |64<br /> |ppn 4<br /> |8.25<br /> |1.50<br /> |----<br /> |8<br /> |64<br /> |ppn 8<br /> |10.71<br /> |1.95<br /> |----<br /> <br /> |}<br /> <br /> Simulations of both 2 and 4 ranks-per-node offered comparable simulation efficiencies to the other full 64 process jobs in this section. The combination of 32 ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. A simulations of ppn=64 with 1 rank-per-node was not successful, and the ppn=32 job required the use of PMEPencils.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |2<br /> |32<br /> |ppn 32<br /> |6.35<br /> |2.29<br /> |----<br /> |2<br /> |32<br /> |ppn 32 (only with PMEPencils 8)<br /> |7.80<br /> |1.95<br /> |----<br /> |4<br /> |32<br /> |ppn 16<br /> |6.30<br /> |2.31<br /> |----<br /> |4<br /> |32<br /> |ppn 16<br /> |10.73<br /> |1.42<br /> |----<br /> |32<br /> |32<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |32<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |32<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |32<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |32<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |32<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5472 2012-11-20T04:53:51Z

<p>Cing: /* Symmetric multiprocessing Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> Comparing 512/ppn4 with 1024/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 2048/ppn4 simulations showed poor scaling so larger systems were not tested.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |2048<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |2048<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |2048<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> In an attempt to maximize the number of processes per node (64), one may also utilize less ranks-per-node, such as 2,4 or 8, while requesting a larger number of threads. The performance of using 8-ranks per node on 64-physical nodes with ppn 8 is approximately equal to the performance of the 16-ranks-per-node above. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |8<br /> |256<br /> |ppn 1<br /> |1.53<br /> |0.56<br /> |----<br /> |8<br /> |256<br /> |ppn 2<br /> |2.77<br /> |1.01<br /> |----<br /> |8<br /> |256<br /> |ppn 4<br /> |4.64<br /> |1.69<br /> |----<br /> |8<br /> |256<br /> |ppn 8<br /> |6.32<br /> |2.30<br /> |----<br /> |8<br /> |512<br /> |ppn 1<br /> |2.79<br /> |0.51<br /> |----<br /> |8<br /> |512<br /> |ppn 2<br /> |5.59<br /> |1.02<br /> |----<br /> |8<br /> |512<br /> |ppn 4<br /> |8.25<br /> |1.50<br /> |----<br /> |8<br /> |512<br /> |ppn 8<br /> |10.71<br /> |1.95<br /> |----<br /> <br /> |}<br /> <br /> Simulations of both 2 and 4 ranks-per-node offered comparable simulation efficiencies to the other full 64 process jobs in this section. The combination of 32 ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. A simulations of ppn=64 with 1 rank-per-node was not successful, and the ppn=32 job required the use of PMEPencils.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |2<br /> |64<br /> |ppn 32<br /> |6.35<br /> |2.29<br /> |----<br /> |2<br /> |128<br /> |ppn 32 (only with PMEPencils 8)<br /> |7.80<br /> |1.95<br /> |----<br /> |4<br /> |128<br /> |ppn 16<br /> |6.30<br /> |2.31<br /> |----<br /> |4<br /> |256<br /> |ppn 16<br /> |10.73<br /> |1.42<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately. PME Pencilsn<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5471 2012-11-20T04:51:19Z

<p>Cing: /* Symmetric multiprocessing Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> Comparing 512/ppn4 with 1024/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 2048/ppn4 simulations showed poor scaling so larger systems were not tested.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |2048<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |2048<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |2048<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> In an attempt to maximize the number of processes per node (64), one may also utilize less ranks-per-node, such as 2,4 or 8, while requesting a larger number of threads. The performance of using 8-ranks per node on 64-physical nodes with ppn 8 is greater than or equal to the performance of the 16-ranks-per-node above. Larger ppn requests may be unstable, and require the use of PMEPencils. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |8<br /> |256<br /> |ppn 1<br /> |1.53<br /> |0.56<br /> |----<br /> |8<br /> |256<br /> |ppn 2<br /> |2.77<br /> |1.01<br /> |----<br /> |8<br /> |256<br /> |ppn 4<br /> |4.64<br /> |1.69<br /> |----<br /> |8<br /> |256<br /> |ppn 8<br /> |6.32<br /> |2.30<br /> |----<br /> |8<br /> |512<br /> |ppn 1<br /> |2.79<br /> |0.51<br /> |----<br /> |8<br /> |512<br /> |ppn 2<br /> |5.59<br /> |1.02<br /> |----<br /> |8<br /> |512<br /> |ppn 4<br /> |8.25<br /> |1.50<br /> |----<br /> |8<br /> |512<br /> |ppn 8<br /> |10.71<br /> |1.95<br /> |----<br /> <br /> |}<br /> <br /> Simulations of both 2 and 4 ranks-per-node offered comparable simulation efficiencies to the other full 64 process jobs in this section. The combination of 32 ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. A simulations of ppn=64 with 1 rank-per-node was not successful.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |2<br /> |64<br /> |ppn 32<br /> |6.35<br /> |2.29<br /> |----<br /> |2<br /> |128<br /> |ppn 32 (only with PMEPencils 8)<br /> |7.80<br /> |1.95<br /> |----<br /> |4<br /> |128<br /> |ppn 16<br /> |6.30<br /> |2.31<br /> |----<br /> |4<br /> |256<br /> |ppn 16<br /> |10.73<br /> |1.42<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5470 2012-11-20T04:48:03Z

<p>Cing: /* Symmetric multiprocessing Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> Comparing 512/ppn4 with 1024/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 2048/ppn4 simulations showed poor scaling so larger systems were not tested.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |2048<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |2048<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |2048<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> In an attempt to maximize the number of processes per node (64), one may also utilize less ranks-per-node, such as 2,4 or 8, while requesting a larger number of threads. The performance of using 8-ranks per node on 64-physical nodes with ppn 8 is greater than or equal to the performance of the 16-ranks-per-node above. Larger ppn requests may be unstable, and require the use of PMEPencils. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |8<br /> |256<br /> |ppn 1<br /> |1.53<br /> |0.56<br /> |----<br /> |8<br /> |256<br /> |ppn 2<br /> |2.77<br /> |1.01<br /> |----<br /> |8<br /> |256<br /> |ppn 4<br /> |4.64<br /> |1.69<br /> |----<br /> |8<br /> |256<br /> |ppn 8<br /> |6.32<br /> |2.30<br /> |----<br /> |8<br /> |512<br /> |ppn 1<br /> |2.79<br /> |0.51<br /> |----<br /> |8<br /> |512<br /> |ppn 2<br /> |5.59<br /> |1.02<br /> |----<br /> |8<br /> |512<br /> |ppn 4<br /> |8.25<br /> |1.50<br /> |----<br /> |8<br /> |512<br /> |ppn 8<br /> |10.71<br /> |1.95<br /> |----<br /> <br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. Simulations larger than ppn 2 were not successful.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |2<br /> |64<br /> |ppn 32<br /> |6.35<br /> |2.29<br /> |----<br /> |2<br /> |128<br /> |ppn 32 (only with PMEPencils 8)<br /> |7.80<br /> |1.95<br /> |----<br /> |4<br /> |128<br /> |ppn 16<br /> |6.30<br /> |2.31<br /> |----<br /> |4<br /> |256<br /> |ppn 16<br /> |10.73<br /> |1.42<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5469 2012-11-19T22:48:17Z

<p>Cing: /* Symmetric multiprocessing Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> Comparing 512/ppn4 with 1024/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 2048/ppn4 simulations showed poor scaling so larger systems were not tested.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |2048<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |2048<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |2048<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> In an attempt to maximize the number of processes per node (64), one may also utilize less ranks-per-node, 8, but specify a larger number of threads. The performance of using 8-ranks per node on 64-physical nodes with ppn 8 is greater than or equal to the performance of the 16-ranks-per-node above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |8<br /> |256<br /> |ppn 1<br /> |1.53<br /> |0.56<br /> |----<br /> |8<br /> |256<br /> |ppn 2<br /> |2.77<br /> |1.01<br /> |----<br /> |8<br /> |256<br /> |ppn 4<br /> |4.64<br /> |1.69<br /> |----<br /> |8<br /> |256<br /> |ppn 8<br /> |6.32<br /> |2.30<br /> |----<br /> |8<br /> |512<br /> |ppn 1<br /> |2.79<br /> |0.51<br /> |----<br /> |8<br /> |512<br /> |ppn 2<br /> |5.59<br /> |1.02<br /> |----<br /> |8<br /> |512<br /> |ppn 4<br /> |8.25<br /> |1.50<br /> |----<br /> |8<br /> |512<br /> |ppn 8<br /> |10.71<br /> |1.95<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. Simulations larger than ppn 2 were not successful.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5468 2012-11-19T20:05:51Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> Comparing 512/ppn4 with 1024/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 2048/ppn4 simulations showed poor scaling so larger systems were not tested.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |2048<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |2048<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |2048<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. Simulations larger than ppn 2 were not successful.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5467 2012-11-19T19:37:44Z

<p>Cing: /* Symmetric multiprocessing (SMP) Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> Comparing 512/ppn4 with 1024/ppn4 we observed 85% scaling, within acceptable criteria for production runs. 2048/ppn4 simulations showed poor scaling so larger systems were not tested.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |2048<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |2048<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |2048<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. Simulations larger than ppn 2 were not successful.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5466 2012-11-19T19:33:40Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Comparing 512/ppn4 with 1024/ppn4 we observed 85% scaling, within acceptable criteria for production runs. Efficiency is respect to the 512/ppn1 simulation.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |16<br /> |2048<br /> |ppn 1<br /> |9.93<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |ppn 2<br /> |13.86<br /> |1.26<br /> |----<br /> |16<br /> |2048<br /> |ppn 3<br /> |14.72<br /> |1.34<br /> |----<br /> |16<br /> |2048<br /> |ppn 4<br /> |15.10<br /> |1.37<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. Simulations larger than ppn 2 were not successful.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5465 2012-11-19T18:40:42Z

<p>Cing: /* Symmetric multiprocessing (SMP) Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% compared to the non-SMP benchmarks. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases. Comparing 512/ppn4 with 1024/ppn4 we observed 85% scaling, within acceptable criteria for production runs. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. Simulations larger than ppn 2 were not successful.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5464 2012-11-19T18:37:48Z

<p>Cing: /* Symmetric multiprocessing (SMP) Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% using 32 physical nodes. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above. Simulations larger than ppn 2 were not successful.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5463 2012-11-19T17:31:27Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% using 32 physical nodes. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. Similarly, combinations of ranks-per-node variation AND ppn flags were unsuccessful with the exception of one simulation. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5462 2012-11-19T17:30:50Z

<p>Cing: /* Performance Tuning Benchmarks */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All ns/day measurements were obtained from the standard output Benchmarking lines within the first few minutes of simulation and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% using 32 physical nodes. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. Similarly, combinations of ranks-per-node variation AND ppn flags were unsuccessful with the exception of one simulation. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5461 2012-11-19T17:30:15Z

<p>Cing: /* Symmetric multiprocessing (SMP) Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All ns/day measurements were obtained from the standard output Benchmarking lines within the first few minutes of simulation and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% using 32 physical nodes. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. Similarly, combinations of ranks-per-node variation AND ppn flags were unsuccessful with the exception of one simulation. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node and ppn 4 as above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5460 2012-11-19T17:29:02Z

<p>Cing: /* Symmetric multiprocessing (SMP) Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All ns/day measurements were obtained from the standard output Benchmarking lines within the first few minutes of simulation and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% using 32 physical nodes. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. Similarly, combinations of ranks-per-node variation AND ppn flags were unsuccessful with the exception of one simulation. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |}<br /> <br /> The combination of increased ranks-per-node and mulithreading was not as effective as using the normal 16 ranks-per-node as above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |}<br /> <br /> PME pencils offered minimal improvements when selected appropriately.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5459 2012-11-19T17:25:21Z

<p>Cing: /* Symmetric multiprocessing (SMP) Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All ns/day measurements were obtained from the standard output Benchmarking lines within the first few minutes of simulation and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% using 32 physical nodes. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. Similarly, combinations of ranks-per-node variation AND ppn flags were unsuccessful with the exception of one simulation. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5458 2012-11-19T17:23:37Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section. All ns/day measurements were obtained from the standard output Benchmarking lines within the first few minutes of simulation and may not represent long-time averages. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following results utilize this feature and offer performance increases of up to 40% using 32 physical nodes. Processors per node (ppn) was varied from 1 (not multithreaded) to 4. Values greater than 4 resulted in a crash. Similarly, combinations of ranks-per-node variation AND ppn flags were unsuccessful with the exception of one simulation. The addition of the Charm++ flag &quot;+CmiNoProcForComThread&quot; to the NAMD binary had negligible improvements in all cases.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.50<br /> |2.00<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.31<br /> |2.30<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.54<br /> |1.01<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31<br /> |1.51<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56<br /> |1.56<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.60<br /> |1.92<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.58<br /> |1.92<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45<br /> |1.62<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.26<br /> |2.27<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88<br /> |1.05<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.34<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35<br /> |2.31<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.24<br /> |2.27<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5457 2012-11-19T17:16:03Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used: nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10. <br /> <br /> The following benchmarks were performed with the non-SMP NAMD build with exception to the last section.<br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Symmetric multiprocessing (SMP) Study ==<br /> All the prior benchmarks were performed with a non-SMP NAMD binary, thus true multithreading was not possible. The following <br /> <br /> {|<br /> |Ranks<br /> |Cores<br /> |Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |ppn 1<br /> |2.75<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2<br /> |4.60<br /> |1.67<br /> |----<br /> |16<br /> |512<br /> |ppn 3<br /> |5.59<br /> |2.03<br /> |----<br /> |16<br /> |512<br /> |ppn 4<br /> |6.31<br /> |2.29<br /> |----<br /> |16<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |2.74<br /> |1.00<br /> |----<br /> |16<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |4.62<br /> |1.68<br /> |----<br /> |16<br /> |512<br /> |ppn 3, +CmiNoProcForComThread<br /> |5.49885623790252<br /> |1.99934563610769<br /> |----<br /> |16<br /> |512<br /> |ppn 4, +CmiNoProcForComThread<br /> |6.30749014454665<br /> |2.29335926412615<br /> |----<br /> |16<br /> |1024<br /> |ppn 1<br /> |5.53831498612652<br /> |1.00684628037535<br /> |----<br /> |16<br /> |1024<br /> |ppn 2<br /> |8.31342729353603<br /> |1.51135198496932<br /> |----<br /> |16<br /> |1024<br /> |ppn 3<br /> |8.56076430503716<br /> |1.55631698798069<br /> |----<br /> |16<br /> |1024<br /> |ppn 4<br /> |10.5920197604721<br /> |1.92559212038466<br /> |----<br /> |16<br /> |1024<br /> |ppn 4, +CmiNoProcForComThread<br /> |10.5836494610629<br /> |1.92407042924813<br /> |----<br /> |32<br /> |512<br /> |ppn 1, +CmiNoProcForComThread<br /> |4.45216813167732<br /> |1.61877716750095<br /> |----<br /> |32<br /> |512<br /> |ppn 2, +CmiNoProcForComThread<br /> |6.25934990391898<br /> |2.27585580961561<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 4<br /> |2.88341765728082<br /> |1.04839047626371<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 16<br /> |6.3429104233047<br /> |2.30623782954063<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 20<br /> |6.35005302294274<br /> |2.30883482877082<br /> |----<br /> |16<br /> |512<br /> |ppn 4, pmePencils 24<br /> |6.23756385705999<br /> |2.26793455548001<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5452 2012-11-16T03:20:13Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used (nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10). <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils). These simulations offered the best performance and were used in production simulations. Note that 1024, the first entry in the table, means that 512 physical cores were requested, but due to the double ranks-per-node, a total of 1024 virtual cores were used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5405 2012-11-05T19:57:04Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used (nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10). <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices and no PME Pencils).<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5404 2012-11-05T19:56:39Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used (nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10). <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively (for the default twoAway choices).<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5403 2012-11-05T19:56:16Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used (nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10). <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |32<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5402 2012-11-05T19:55:44Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. The following non-bonded frequency parameters were also used (nonbondedFreq=1, fullElectFrequency=2, stepspercycle=10). <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |16<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5401 2012-11-05T19:51:47Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes respectively.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |16<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5400 2012-11-05T19:50:22Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.43<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.3<br /> |----<br /> |16<br /> |8192<br /> |<br /> |19.81<br /> |1.12<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5399 2012-11-05T19:41:32Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. The following efficiency estimates are measured with respect to the 16 ranks per node results for the same number of nodes.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.44<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.23<br /> |----<br /> |16<br /> |8192<br /> |<br /> |19.81<br /> |0.89<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5398 2012-11-05T19:38:41Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 due to memory errors, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.44<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.23<br /> |----<br /> |16<br /> |8192<br /> |<br /> |19.81<br /> |0.89<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5397 2012-11-05T19:37:54Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64 for all simulations, and also resulted in out of memory errors for 16384 core simulations of 32 ranks per node. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.44<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.23<br /> |----<br /> |16<br /> |8192<br /> |<br /> |19.81<br /> |0.89<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5396 2012-11-05T19:29:12Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. In this study, memory requirements were too large to use 64, and resulted in out of memory errors for 16384 core simulations of 32 ranks per node. <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |32<br /> |1024<br /> |<br /> |4.46<br /> |1.6<br /> |----<br /> |32<br /> |2048<br /> |<br /> |8.01<br /> |1.44<br /> |----<br /> |32<br /> |4096<br /> |<br /> |13.74<br /> |1.23<br /> |----<br /> |16<br /> |8192<br /> |<br /> |19.81<br /> |0.89<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5378 2012-10-30T04:03:17Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64. Benchmarks to be updated for this section...<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5377 2012-10-30T00:11:36Z

<p>Cing: /* Incorrect Particle-Mesh Ewald Grid */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero, with no measurable effect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect with none of the grid dimensions are divisible by 5. One can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5376 2012-10-29T23:47:53Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms and the simulation time-step was 2fs. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero, with no measurable effect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect where one can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5375 2012-10-29T22:33:39Z

<p>Cing: /* Incorrect Particle-Mesh Ewald Grid */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero, with no measurable effect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect where one can draw a comparison to a more correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5374 2012-10-29T22:33:22Z

<p>Cing: /* Incorrect Particle-Mesh Ewald Grid */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero, with no measurable effect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect where one can draw a comparison to the correct PME choice in the Performance Tuning Benchmarks above.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5373 2012-10-29T22:31:23Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero, with no measurable effect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5372 2012-10-29T22:29:38Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc). The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5371 2012-10-29T22:29:17Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster, [[BGQ]], with attention to NAMD performance tuning documentation. Determining optimal parameters for a NAMD simulation on this system is more difficult as there are only certain simulation sizes that have optimal topologies (512, 1024, etc.) The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5370 2012-10-29T22:26:33Z

<p>Cing: /* Performance Tuning Benchmarks */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three &quot;'''Benchmark time'''&quot; lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5369 2012-10-29T22:26:17Z

<p>Cing: /* Incorrect Particle-Mesh Ewald Grid */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three '''Benchmark time''' lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the &quot;'''pmeGridSpacing 1.0'''&quot; setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5368 2012-10-29T22:26:01Z

<p>Cing: /* Ranks-Per-Node Study */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three '''Benchmark time''' lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency. Additionally, one study was conducted with the flag &quot;'''ldbUnloadZero yes'''&quot; which supposedly ensures no work is scheduled for processor zero.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5367 2012-10-29T22:24:41Z

<p>Cing: /* PME Pencils */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three '''Benchmark time''' lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5366 2012-10-29T22:22:07Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three '''Benchmark time''' lines at the beginning of the simulation's standard output. In this section, the PME patch grid was manually doubled in either the X, Y, or Z directions. Default PME patch doubling in NAMD 2.9 is generally recommended (twoAway parameters need not be specified in the configuration file). <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> == PME Pencils ==<br /> A &quot;pencil-based&quot; PME decomposition may be more efficient than the default &quot;slab-based decomposition&quot;. In this study PME pencil grids are created for both dedicated PME nodes (lblUnload=yes benchmarks) and non-dedicated PME nodes. Fine-tuning of PMEPencils resulted in insignificant performance gains for this study.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |}<br /> <br /> == Ranks-Per-Node Study ==<br /> The &quot;ranks-per-node&quot; or simply the number of processes per compute node is a Blue Gene/Q runjob command parameter. For some simulations increasing the number of processes beyond the normal 16 may be more effective. In this study, memory requirements were too large to use 64 and decreased performance using 32. Ranks per node of less than 16 were also tested, although this would be less than optimal for cluster efficiency.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5365 2012-10-29T22:08:13Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three '''Benchmark time''' lines at the beginning of the simulation's standard output and any additional configuration options are listed where they are used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |}<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5364 2012-10-29T22:07:18Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency is measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started using a restart file from a pre-equilibrated snapshot. Performance in nanoseconds per day is based on the geometric mean of the three '''Benchmark time''' lines at the beginning of the simulation's standard output and any additional configuration options are listed where they are used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5363 2012-10-29T22:03:22Z

<p>Cing: /* Incorrect Particle-Mesh Ewald Grid */</p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency us measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started from a pre-equilibrated snapshots restart file. Performance in nanoseconds per day is based on the geometric mean of the three '''Benchmark time''' lines at the beginning of the simulation's standard output and any additional configuration options are listed where they are used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations above with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation due to the matrix size requirements in the FFT algorithm. Below is an example of the type of performance degradation that one may expect.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5362 2012-10-29T21:56:34Z

<p>Cing: </p> <hr /> <div>A parameter study was undertaken to test simulation performance and efficiency of NAMD on the Blue Gene/Q cluster with attention to NAMD performance tuning documentation. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency us measured with respect to the 16 ranks-per-node 512 core simulation. All simulations are started from a pre-equilibrated snapshots restart file. Performance in nanoseconds per day is based on the geometric mean of the three '''Benchmark time''' lines at the beginning of the simulation's standard output and any additional configuration options are listed where they are used.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5361 2012-10-29T21:51:49Z

<p>Cing: </p> <hr /> <div>Taking into consideration the NAMD performance tuning documentation, a parameter study was undertaken to test simulation performance and efficiency of NAMD on the BG/Q cluster. The system of study is a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. <br /> <br /> == Performance Tuning Benchmarks ==<br /> <br /> Efficiency was measured with respect to the 16 ranks-per-node 512 core simulation. All simulations were conducted from a restart trajectory from an equilibrated system.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Incorrect Particle-Mesh Ewald Grid ==<br /> <br /> Long-range electrostatics are computed using PME for all simulations with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in increasingly large performance degradation.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |Poor PME Multiple (144x144x111)<br /> |2.70<br /> |0.97<br /> |----<br /> |16<br /> |1024<br /> |Poor PME Multiple (144x144x111)<br /> |5.13<br /> |0.92<br /> |----<br /> |16<br /> |2048<br /> |Poor PME Multiple (144x144x111)<br /> |8.61<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |Poor PME Multiple (144x144x111)<br /> |13.93<br /> |0.62<br /> |----<br /> |16<br /> |8192<br /> |Poor PME Multiple (144x144x111)<br /> |17.08<br /> |0.38<br /> |----<br /> |16<br /> |16384<br /> |Poor PME Multiple (144x144x111)<br /> |17.64<br /> |0.20<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5360 2012-10-29T21:38:57Z

<p>Cing: </p> <hr /> <div>Taking into consideration the official (test)[NAMD performance tuning recommendations], a parameter study was undertaken to test simulation performance and efficiency on the BG/Q cluster. Efficiency was measured with respect to the 16 ranks-per-node 512 core simulation. All simulations were conducted from a restart trajectory from an equilibrated system.<br /> <br /> The following study was undertaken for a 246,000 atom membrane protein simulation ([http://www.rcsb.org/pdb/explore.do?structureId=1m56 Cytochrome c Oxidase] embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). The unit cell is cubic with box dimensions 144 x 144 x 117 Angstroms. Long-range electrostatics are computed using PME for all simulations with PME grid spacing set to be generated automatically with the '''pmeGridSpacing 1.0''' setting. A poor choice in PME grid spacing (i.e. not a multiple of 2,3, and 5) can result in performance degradation of <br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.05<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.62<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.07<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.59<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.32<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.63<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.79<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.52<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |25.00<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.67<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.31<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.98<br /> |0.31<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.93<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.27<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.02<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.41<br /> |0.69<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.21<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.92<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.83<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.97<br /> |0.76<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.24<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.99<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.94<br /> |0.63<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.86<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.84<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.79<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.29<br /> |0.82<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |512<br /> |ldbUnloadZero=yes<br /> |2.79<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5359 2012-10-29T21:26:45Z

<p>Cing: </p> <hr /> <div>Taking into consideration the official (test)[NAMD performance tuning recommendations], a parameter study was undertaken to test simulation performance and efficiency on the BG/Q cluster. <br /> <br /> The following study was undertaken for a 246,000 atom membrane protein (Cytochrome c Oxdaise embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). Efficiency was measured with respect to the 16 ranks-per-node 512 core simulation.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.7874<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.0483<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.6204<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.0699<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.5875<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.3285<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.6313<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.7962<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.5198<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |24.9981<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.6706<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.3096<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.9791<br /> |0.31<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=8, lblUnload=yes<br /> |12.9322<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12, lblUnload=yes<br /> |17.2722<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16, lblUnload=yes<br /> |16.0208<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20, lblUnload=yes<br /> |15.4154<br /> |0.69<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=12<br /> |16.2079<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=16<br /> |17.9231<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.9949<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=24<br /> |17.8308<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=36<br /> |16.9666<br /> |0.76<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |18.2473<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |17.9949<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, PMEPencils=20<br /> |13.9414<br /> |0.63<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.8589<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.8367<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.7874<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.2951<br /> |0.82<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |512<br /> |unloadZero<br /> |2.7932<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing https://oldwiki.scinet.utoronto.ca/index.php?title=Namd_on_BGQ&diff=5358 2012-10-29T21:25:24Z

<p>Cing: Created page with &quot;Taking into consideration the official (test)[NAMD performance tuning recommendations], a parameter study was undertaken to test simulation performance and efficiency on the B...&quot;</p> <hr /> <div>Taking into consideration the official (test)[NAMD performance tuning recommendations], a parameter study was undertaken to test simulation performance and efficiency on the BG/Q cluster. <br /> <br /> The following study was undertaken for a 246,000 atom membrane protein (Cytochrome c Oxdaise embedded in a TIP3P solvated DPPC bilayer) using the CHARMM36 forcefield (protein and lipids). Efficiency was measured with respect to the 16 ranks-per-node 512 core simulation.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |Ranks<br /> |Cores<br /> |NAMD Config Options<br /> |ns/day<br /> |Efficiency<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.7874<br /> |1.00<br /> |----<br /> |16<br /> |1024<br /> |<br /> |5.0483<br /> |0.91<br /> |----<br /> |16<br /> |1024<br /> |twoAwayX (default)<br /> |5.6204<br /> |1.01<br /> |----<br /> |16<br /> |2048<br /> |twoAwayX (default)<br /> |10.0699<br /> |0.90<br /> |----<br /> |16<br /> |2048<br /> |twoAwayXY<br /> |10.5875<br /> |0.95<br /> |----<br /> |16<br /> |4096<br /> |twoAwayX<br /> |14.3285<br /> |0.64<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY (default)<br /> |17.6313<br /> |0.79<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXYZ<br /> |16.7962<br /> |0.75<br /> |----<br /> |16<br /> |8192<br /> |twoAwayX<br /> |23.5198<br /> |0.53<br /> |----<br /> |16<br /> |8192<br /> |twoAwayXY (default)<br /> |24.9981<br /> |0.56<br /> |----<br /> |16<br /> |16384<br /> |twoAwayX<br /> |23.6706<br /> |0.27<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXY<br /> |28.3096<br /> |0.32<br /> |----<br /> |16<br /> |16384<br /> |twoAwayXYZ (default)<br /> |27.9791<br /> |0.31<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils8, lblUnload=yes<br /> |12.9322<br /> |0.58<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils12, lblUnload=yes<br /> |17.2722<br /> |0.77<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils16, lblUnload=yes<br /> |16.0208<br /> |0.72<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils20, lblUnload=yes<br /> |15.4154<br /> |0.69<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils12<br /> |16.2079<br /> |0.73<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils16<br /> |17.9231<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils20<br /> |17.9949<br /> |0.81<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils24<br /> |17.8308<br /> |0.80<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils36<br /> |16.9666<br /> |0.76<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |8<br /> |4096<br /> |twoAwayXY, Pencils20<br /> |18.2473<br /> |0.82<br /> |----<br /> |16<br /> |4096<br /> |twoAwayXY, Pencils20<br /> |17.9949<br /> |0.81<br /> |----<br /> |32<br /> |4096<br /> |twoAwayXY, Pencils20<br /> |13.9414<br /> |0.63<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |4<br /> |512<br /> |<br /> |2.8589<br /> |1.03<br /> |----<br /> |8<br /> |512<br /> |<br /> |2.8367<br /> |1.02<br /> |----<br /> |16<br /> |512<br /> |<br /> |2.7874<br /> |1.00<br /> |----<br /> |32<br /> |512<br /> |<br /> |2.2951<br /> |0.82<br /> |----<br /> |<br /> |<br /> |<br /> |<br /> |<br /> |----<br /> |16<br /> |512<br /> |unloadZero<br /> |2.7932<br /> |1.00<br /> |----<br /> |}<br /> <br /> == Documentation ==<br /> <br /> # [http://www.ks.uiuc.edu/Research/namd/2.9/ug/node39.html NAMD 2.9 User Guide]<br /> # [http://www.ks.uiuc.edu/Research/namd/wiki/?NamdPerformanceTuning NAMD Performance Tuning Wiki]</div>

Cing