Running CCSM3

You can run CCSM3 in interactive mode on GPC without a host file, but only do so for very short compilation/initialization tests. Run the model in the TCS or GPC batch queue.

Create a setup script:

#!/bin/csh

setenv CCSMROOT /project/ccsm/ccsm3_current
setenv SCRATCH /scratch/$USER
setenv CASEROOT /home/$USER/runs/ccsm3_t31_gpc
setenv MACH gpc
cd $CCSMROOT/scripts
./create_newcase -mach $MACH -res T31_gx3v5 -case $CASEROOT

Then:

cd /home/$USER/runs/ccsm3_t31_gpc

Modify env_mach.gpc, env_run, env_conf to your needs.

Configure and Build:

./configure -mach gpc
./ccsm3_t31_gpc.gpc.build

Run

./ccsm3_t31_gpc.gpc.run (for interactive)
 or qsub ccsm3_t31_gpc.gpc.run

To Run on TCS

./ccsm3_t31_gpc.gpc.run (for interactive)
or llsubmit ccsm3_t31_tcs.tcs.run

To Run interactively on TCS:

Set up env_mach.tcs to have the correct number of processors (i.e. less then the 64 threads on the node, e.g. 16).

after you configure and build, add some environment variables to the .run script (MPI_HOSTFILE, MP_PROCS, MP_NODES)

#-----------------------------------------------------------------------
# Determine necessary environment variables
#----------------------------------------------------------------------- 

cd /project/peltier/guido/runs/T42_tcs 
setenv MACH tcs
setenv PATH /scratch/$LOGNAME/bin:$PATH
setenv MP_EUIDEVICE sn_all
#setenv LAPI_DEBUG_SLOT_ATT_THRESH 500000
setenv MP_RC_USE_LMC yes
setenv MP_POLLING_INTERVAL 20000000
setenv MP_EAGER_LIMIT 65536
setenv MP_BULK_MIN_MSG_SIZE 65537
setenv MPI_HOSTFILE /project/peltier/guido/runs/T42_tcs/hostfile
setenv MP_PROCS 16
setenv MP_NODES 1
source env_conf || "problem sourcing env_conf" && exit -1  
source env_run  || "problem sourcing env_run" && exit -1  
source env_mach.tcs || "problem sourcing env_mach.tcs" && exit -1

change

timex poe -cmdfile poe.cmdfile

to

timex poe -cmdfile poe.cmdfile -hfile /project/peltier/guido/runs/T42_tcs/hostfile

Then run interactively (e.g. ./YOURRUNNAME.run , then use top in another terminal to see it run)

To Run interactively on GPC:

Set up env_mach.gpc to have the correct number of processors (i.e. less then the 8 tasks per node time the nodes , e.g. 16).

Compile the model on one of the 4 GPC interactive nodes

In env_run in your run script directory, change:

setenv SETBLD        AUTO            # [AUTO, TRUE, FALSE]

to

setenv SETBLD        FALSE            # [AUTO, TRUE, FALSE]

Then get on the interactive nodes

qsub -I -l nodes=2:ppn=8,walltime=1:00:00

and run interactively:

./YOURRUNNAME.run

NOTE: Both CCSM3 and CCSM4/CESM1 can be run on the GPC batch queue by instating the full environment setup

Add "module load extras" to your env_machopts (h/t Chris)