Installing CCSM3

This page is mostly a record of the steps needed to install CCSM3 on the General Purpose Cluster (GPC) (Linux). This may be useful for running the model through it's initialization step, if for example, you want to change the boundary conditions and need to see if the model initializes with the new boundary conditions.

The first several lines of the Macros.Linux (configuration) file (Using Intel Fortran Compilers):

#===============================================================================
# CVS $Id: Macros.Linux,v 1.11.2.4 2007/01/17 05:17:49 tcraig Exp $
# CVS $Source: /fs/cgd/csm/models/CVS.REPOS/shared/bld/Macros.Linux,v $
# CVS $Name: ccsm3_0_1_beta24 $
#===============================================================================
### Makefile macros for "Linux", supports portland + gnu 
# Makefile macros for "Linux", supports Intel compilers + gnu 
#===============================================================================  

INCLDIR    := -I. -I$(FPATH) -I$(SCINET_NETCDF_INC) -I$(INCROOT) -I$(INC_MPI) -I$(NETCDF_MOD)

#SLIBS      := -L$(LIB_NETCDF) -lnetcdf  -llapack -lblas
#SLIBS      := -L$(LIBRARY_PATH) -lmkl_lapack -lmkl_em64t -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread -L/usr/local/lib -lnetcdf
SLIBS      := -L$(SCINET_NETCDF_LIB) -lnetcdf

CPP        := NONE
CPPFLAGS   :=
CPPDEFS    := -DLINUX -DFORTRANUNDERSCORE -DLINUX  -DNO_SHR_VMATH
CPPFLAGS   := -DLINUX -DNO_SHR_VMATH -DINTEL_COMPILER -Df2cFortran
FC         := mpif90
#FFLAGS     := -c -r8 -i4 -Kieee -Mrecursive -Mdalign -Mextend
#FFLAGS     := -c -real-size 64 -integer-size 32 -align all -fltconsistency -recursive -extend_source 132
## This compiles on intel x86_64
FFLAGS     := -O0 -cpp -c -real-size 64 -integer-size 32 -align all -fltconsistency -save   

#FFLAGS     := -c -r8 -i4 -autodouble -align all -fltconsistency -recursive -fast
#FFLAGS     := -cpp -c -r8 -i4 -132 -autodouble -convert big_endian -fp-model precise -prec-div -prec-sqrt -recursive -align all -fltconsistency 
CC         := mpicc
CFLAGS     := -c -DUSE_GCC

FIXEDFLAGS := 
FREEFLAGS  := -free
MOD_SUFFIX := mod
LD         := $(FC)
AR         := ar
ULIBS      := -L$(LIBROOT) -lesmf -lmct -lmpeu -lmph
FLIBS      := -L/usr/local/lib

In /project/ccsm/ccsm3_current/scripts/ccsm_utils/Machines

Modified these files:

run.linux.gpc added:

mpirun -np $NTASKS[1] ./$COMPONENTS[1] : -np $NTASKS[2] ./$COMPONENTS[2] : -np $NTASKS[3] ./$COMPONENTS[3] : -np $NTASKS[4] ./$COMPONENTS[4] : -np $NTASKS[5] ./$COMPONENTS[5]

batch.linux.gpc Mirrored the other linux batch config file, you may need to change in your run script after the model is built to run on torque PBS properly

#! /bin/csh -f
#===============================================================================
#  This is a CCSM batch job script for $mach
#===============================================================================
## BATCH INFO
#PBS -N ${jobname}
#PBS -q ${qname}
##PBS -l nodes=${nodes}:ib:ppn=${tasks}
#PBS -l walltime=${tlimit}
#PBS -r n
#PBS -j oe
#PBS -k oe
#PBS -S /bin/csh -V
 
limit coredumpsize 1000000
limit stacksize unlimited

env.linux.gpc

# General machine specific environment variables  - edit before the initial build
# -------------------------------------------------------------------------

setenv LIB_NETCDF $SCINET_NETCDF_LIB
setenv INC_NETCDF $SCINET_NETCDF_INC
setenv INC_MPI $SCINET_MPI_INC
setenv SCRATCH /scratch/$USER/

if !($?SCRATCH) then
  set SCRATCH = $HOME
  echo "## Warning: SCRATCH not defined in system environment. Set SCRATCH to be $HOME";
endif

setenv EXEROOT             $SCRATCH/exe/$CASE
setenv RUNROOT             $EXEROOT
setenv GMAKE_J             1 

#####set the path for netcdf module: netcdf.mod
if(! $?NETCDF_MOD) then
   ##### user need to set this variable.
   ### setenv NETCDF_MOD /usr/local/lib64/r4i4
endif


# -------------------------------------------------------------------------
# Environment variables for prestaging input data - edit anytime during run
# -------------------------------------------------------------------------  

setenv DIN_LOC_ROOT        /project/ccsm/inputdata
setenv DIN_LOC_ROOT_USER   /project/ccsm/inputdata_user

Modified check_machine to include gpc:

/project/ccsm/ccsm3_current/scripts/ccsm_utils/Tools/check_machine

Changed:

/project/ccsm/ccsm3_current/scripts/ccsm_utils/Components/esmf.buildlib

diff esmf.buildlib esmf.buildlib~

24c24
< if ($OS == 'Linux')        setenv ESMF_ARCH linux_intel
---
> if ($OS == 'Linux')        setenv ESMF_ARCH linux_pgi

Edit /project/ccsm/ccsm3_current/scripts/ccsm_utils/Components/mct.buildlib

MCT configure wants to grab gfortran because of the PATH order

if ( `uname` == "Linux" ) then
    setenv FC mpif90
endif

Finally make sure the following modules are loaded: intel, intelmpi, netcdf4.0.1_nc3, gcc

I have the following set on GPC:

 Currently Loaded Modulefiles:

 1) intel/intel-v11.1.072     3) gcc/4.4.0                 5) netcdf/4.0.1_nc3_intel    7) parallel-netcdf/1.1.1
 2) intelmpi/impi-4.0.0.027   4) Xlibraries/X11-64         6) python/2.6.2

My .bashrc has:

if [ "${HOST}" == "AIX" ]; then

 # do things for the TCS machine
 :

else

 # do things for the GPC machine
 module load intel intelmpi gcc Xlibraries netcdf python parallel-netcdf
 export MACH="gpc"
 export PATH="/home/$LOGNAME/bin:$PATH:/scinet/gpc/bin"
 :

fi

My PATH is: /home/guido/bin:/scinet/gpc/Libraries/parallel-netcdf-1.1.1/bin:/scinet/gpc/tools/Python/Python262/bin:/scinet/gpc/compilers/gcc/bin/:/scinet/gpc/intel/impi/4.0.0.027/bin64/:/scinet/gpc/intel/Compiler/11.1/072/bin/intel64/:/usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/lpp/mmfs/bin:/opt/torque/bin:/opt/torque/sbin:/usr/lpp/mmfs/bin:/opt/torque/bin:/opt/torque/sbin:/scinet/gpc/x11/bin:/scinet/gpc/Libraries/netcdf-4.0.1_nc3_intel/bin:/scinet/gpc/bin