Gamess

From oldwiki.scinet.utoronto.ca
Revision as of 14:40, 1 December 2011 by Dgruner (talk | contribs)
Jump to navigation Jump to search

GAMESS(US)

GAMESS version August 18, 2011, R1, on CentOS6

The current version of GAMESS(US), August 18, 2011 R1, was built on CentOS6, using the Intel v12.1 compilers and IntelMPI v4.0.2.003 MPI library. The build was done according to the documentation that came with the package, available on the SciNet systems in the /scinet/gpc/src/gamess-Aug1811R1 directory. After running "./config", with the intel and intelmpi modules loaded, the following configuration file (install.info) was obtained: 

  [dgruner@gpc-f103n058 gamess-Aug1811R1]$ cat install.info
  #!/bin/csh
  #   compilation configuration for GAMESS
  #   generated on gpc-f103n058
  #   generated at Wed Nov 30 11:18:29 EST 2011
  setenv GMS_PATH            /home/scinet/gpc/src/gamess-Aug1811R1
  #         machine type
  setenv GMS_TARGET          linux64
  #         FORTRAN compiler setup
  setenv GMS_FORTRAN         ifort
  setenv GMS_IFORT_VERNO     12
  #         mathematical library setup
  setenv GMS_MATHLIB         mkl
  setenv GMS_MATHLIB_PATH    /scinet/gpc/intel/ics/composer_xe_2011_sp1.6.233/mkl/lib/intel64
  setenv GMS_MKL_VERNO       "12-so"
  #         parallel message passing model setup
  setenv GMS_DDI_COMM        mpi
  setenv GMS_MPI_LIB         impi
  setenv GMS_MPI_PATH        /scinet/gpc/intel/impi/4.0.2.003


The code was built with the "compall" script, and then linked using the "lked" script. A new module, gamess/Aug1811R1, was created for CentOS6: 

  [dgruner@gpc-f103n058 gamess-Aug1811R1]$ module show gamess
  -------------------------------------------------------------------
  /scinet/gpc/Modules6/Modules/modulefiles/gamess/Aug1811R1:
  
  module-whatis	 adds GAMESS (US), Aug 18/11 R1, environment variables 
  prereq    intel 
  prereq    intelmpi 
  setenv    SCINET_GAMESS_HOME /scinet/gpc/Applications/gamess-Aug1811R1 
  setenv    SCINET_RUNGMS /scinet/gpc/Applications/gamess-Aug1811R1/rungms 
  setenv    SCINET_GAMESS_SCRATCH /scratch/s/scinet/dgruner/gamess-scratch 
  -------------------------------------------------------------------

(note that the $SCINET_GAMESS_SCRATCH directory is defined for each user).

The $SCINET_RUNGMS run script will work with both ethernet and infiniband connected nodes, so unlike the older version there is no need for specialized scripts depending on the node interconnect. Here are some sample scripts showing how to run gamess on the GPC:

Running GAMESS

- Make sure the directory $SCRATCH/gamess-scratch exists (the $SCINET_RUNGMS script will create it if it does not exist)

- Make sure the modules: intel, intelmpi, gamess are loaded (in your .bashrc: "module load intel intelmpi gamess").

- Create a torque script to run GAMESS, as in the examples that follow

- The GAMESS executable is in $SCINET_GAMESS_HOME/gamess.00.x - The rungms script is in $SCINET_GAMESS_HOME/rungms (also defined as $SCINET_RUNGMS)

- The rungms script takes 4 arguments: input file, executable number, number of compute processes, elongations. It always assumes there are 8 processors per node (ppn). Note that you can copy the $SCINET_RUNGMS script to your own directory, and - carefully - modify it to suit your needs (at your own risk!).

For example, in order to run with the input file /scratch/$USER/gamesstest01, on 8 cpus, and the default version (00) of the executable on a machine with 8 cores: 

  # load the gamess and other required modules in .bashrc
  module load intel intelmpi gamess  

  # run the program
  $SCINET_RUNGMS $SCRATCH/gamesstest01 00 8

Here is a sample torque script for running a GAMESS calculation, on a single 8-core node. Note the comments regarding the use of the ramdisk, a technique that may speed up calculations significantly. It is restricted to runs on a single node, and there is book-keeping required before the job ends, namely copying any useful output from the ramdisk (/dev/shm) to the user's directory on the /scratch filesystem.

<source lang="bash">

  1. !/bin/bash
  2. PBS -l nodes=1:ppn=8,walltime=48:00:00,os=centos6computeA
  3. PBS -N gamessjob
  1. If not an interactive job (i.e. -I), then cd into the directory where
  2. I typed qsub.

if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then 

  if [ -n "$PBS_O_WORKDIR" ]; then
    cd $PBS_O_WORKDIR
  fi

fi

  1. the input file is typically named something like "gamesjob.inp"
  2. so the script will be run like "$SCINET_RUNGMS gamessjob 00 8"
  1. load the gamess module if not in .bashrc already
  2. module load intel intelmpi gamess
  1. If - and only if - the run is on a single node, and does not require too much memory,
  2. then the runtime scratch directory can be put on ramdisk on the
  3. compute node. This helps speed up the code, and avoids contention for
  4. disk resources. Note that the maximum size of the ramdisk is 8 GB,
  5. and the maximum total memory available to jobs is about 14 GB per node
  6. (including program memory and ramdisk).
  8. In order to use the ramdisk uncomment the following definition:
  9. export SCINET_GAMESS_SCRATCH=/dev/shm
  1. run the program

$SCINET_RUNGMS gamessjob 00 8

  1. copy any files of interest from the $SCINET_GAMESS_SCRATCH directory
  2. to a "real" directory on disk, e.g. /scratch/$USER/gamessjob
  3. cp $SCINET_GAMESS_SCRATCH/* /scratch/$USER/gamessjob

</source>


Here is a similar script, but using 2 nodes with infiniband interconnect:

<source lang="bash">

  1. !/bin/bash
  2. PBS -l nodes=2:ib:ppn=8,walltime=48:00:00,os=centos6computeA
  3. PBS -N gamessjob
    1. To submit type: qsub gmsib.sh
  1. If not an interactive job (i.e. -I), then cd into the directory where
  2. I typed qsub.

if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then 

  if [ -n "$PBS_O_WORKDIR" ]; then
    cd $PBS_O_WORKDIR
  fi

fi

  1. the input file is typically named something like "gamesjob.inp"
  2. so the script will be run like "$SCINET_RUNGMS gamessjob 00 8 8"
  1. load the gamess module if not in .bashrc alreadyc

module load intel intelmpi gamess

  1. This script requests InfiniBand-connected nodes (:ib above). The IntelMPI should run with the best transport it can find (InfiniBand), and will default back to ethernet if running on ethernet-only nodes.

$SCINET_RUNGMS gamessjob 00 16 </source>



Notes on older version, May 22, 2009, built on CentOS5

The GAMESS version January 12, 2009 R3 was built using the Intel v11.1 compilers and v3.2.2 MPI library, according to the instructions in http://software.intel.com/en-us/articles/building-gamess-with-intel-compilers-intel-mkl-and-intel-mpi-on-linux/

The required build scripts - comp, compall, lked - and run script - rungms - were modified to account for our own installation. In order to build GAMESS one first must ensure that the intel and intelmpi modules are loaded ("module load intel intelmpi"). This applies to running GAMESS as well. The module "gamess" must also be loaded in order to run GAMESS ("module load gamess").

The modified scripts are in the file /scinet/gpc/src/gamess-on-scinet.tar.gz

Running GAMESS

- Make sure the directory /scratch/$USER/gamess-scratch exists (the $SCINET_RUNGMS script will create it if it does not exist)

- Make sure the modules: intel, intelmpi, gamess are loaded (in your .bashrc: "module load intel intelmpi gamess").

- Create a torque script to run GAMESS. Here is an example:

- The GAMESS executable is in $SCINET_GAMESS_HOME/gamess.00.x - The rungms script is in $SCINET_GAMESS_HOME/rungms (actually it is $SCINET_RUNGMS)

- For multinode runs, use the $SCINET_RUNGMS_IB script to run on the InfiniBand portion of the GPC

- The rungms script takes 4 arguments: input file, executable number, number of compute processes, processors per node

For example, in order to run with the input file /scratch/$USER/gamesstest01, on 8 cpus, and the default version (00) of the executable on a machine with 8 cores: 

  # load the gamess and other required modules in .bashrc
  module load gcc intel intelmpi gamess  

  # run the program
  $SCINET_RUNGMS /scratch/$USER/gamesstest01 00 8 8

Here is a sample torque script for running a GAMESS calculation, on a single 8-core node. Note the comments regarding the use of the ramdisk, a technique that may speed up calculations significantly. It is restricted to runs on a single node, and there is book-keeping required before the job ends, namely copying any useful output from the ramdisk (/dev/shm) to the user's directory on the /scratch filesystem.

<source lang="bash">

  1. !/bin/bash
  2. PBS -l nodes=1:ppn=8,walltime=48:00:00,os=centos53computeA
  3. PBS -N gamessjob
    1. To submit type: qsub gms.sh
  1. If not an interactive job (i.e. -I), then cd into the directory where
  2. I typed qsub.

if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then 

  if [ -n "$PBS_O_WORKDIR" ]; then
    cd $PBS_O_WORKDIR
  fi

fi

  1. the input file is typically named something like "gamesjob.inp"
  2. so the script will be run like "$SCINET_RUNGMS gamessjob 00 8 8"
  1. load the gamess module if not in .bashrc already
  2. module load gcc intel intelmpi gamess
  1. If - and only if - the run is on a single node, and does not require too much memory,
  2. then the runtime scratch directory can be put on ramdisk on the
  3. compute node. This helps speed up the code, and avoids contention for
  4. disk resources. Note that the maximum size of the ramdisk is 8 GB,
  5. and the maximum total memory available to jobs is about 14 GB per node
  6. (including program memory and ramdisk).
  8. In order to use the ramdisk uncomment the following definition:
  9. export SCINET_GAMESS_SCRATCH=/dev/shm
  1. run the program

$SCINET_RUNGMS gamessjob 00 8 8

  1. copy any files of interest from the $SCINET_GAMESS_SCRATCH directory
  2. to a "real" directory on disk, e.g. /scratch/$USER/gamessjob
  3. cp $SCINET_GAMESS_SCRATCH/* /scratch/$USER/gamessjob

</source>

Here is a similar script, but this one uses 2 InfiniBand-connected nodes, and runs the appropriate $SCINET_RUNGMS_IB script to actually run the job:

<source lang="bash">

  1. !/bin/bash
  2. PBS -l nodes=2:ib:ppn=8,walltime=48:00:00,os=centos53computeA
  3. PBS -N gamessjob
    1. To submit type: qsub gmsib.sh
  1. If not an interactive job (i.e. -I), then cd into the directory where
  2. I typed qsub.

if [ "$PBS_ENVIRONMENT" != "PBS_INTERACTIVE" ]; then 

  if [ -n "$PBS_O_WORKDIR" ]; then
    cd $PBS_O_WORKDIR
  fi

fi

  1. the input file is typically named something like "gamesjob.inp"
  2. so the script will be run like "$SCINET_RUNGMS gamessjob 00 8 8"
  1. load the gamess module if not in .bashrc already
  2. actually, it MUST be in .bashrc
  3. module load gamess
  1. This script requests InfiniBand-connected nodes (:ib above)
  2. so it must run with the IB version of the rungms script,
  3. $SCINET_RUNGMS_IB

$SCINET_RUNGMS_IB gamessjob 00 16 8 </source>


-- dgruner 5 October 2009 -- dgruner Updated on 14 September 2010

Retrieved from ‘https://oldwiki.scinet.utoronto.ca/index.php?title=Gamess&oldid=4220’