Running CCSM4

The scientifically validated CCSM4 runs are found in the list below (including a complete list of the model resolutions):

CCSM has significant flexibility to configure cases with respect to components, grids, and model settings. This version of CCSM has only be validated scientifically for the following full active configurations:

• 1.9x2.5_gx1v6 B_1850_CN
• 1.9x2.5_gx1v6 B_1850_RAMPCO2_CN
• 0.9x1.25_gx1v6 B_1850_CN
• 0.9x1.25_gx1v6 B_1850_RAMPCO2_CN

If the user is interested in running a "stand-alone" component configuration, the following model configurations have been validated scientifically and have associated diagnostic output as part of the release.

• 0.9x1.25_gx1v6 F_AMIP_1DEG
• 1.9x2.5_gx1v6 F_AMIP_2DEG
• 0.9x1.25_gx1v6 I and ICN
• T62_gx1v6 C

Please refer to the individual component release web pages above for information regarding alternative component configurations.

• pt1_pt1 (SHORTNAME: pt1)
• 0.47x0.63_0.47x0.63 (SHORTNAME: f05_f05)
• 0.47x0.63_gx1v6 (SHORTNAME: f05_g16)
• 0.47x0.63_tx0.1v2 (SHORTNAME: f05_t12)
• 0.9x1.25_0.9x1.25 (SHORTNAME: f09_f09)
• 0.9x1.25_gx1v6 (SHORTNAME: f09_g16)
• 1.9x2.5_1.9x2.5 (SHORTNAME: f19_f19)
• 1.9x2.5_gx1v6 (SHORTNAME: f19_g16)
• 4x5_4x5 (SHORTNAME: f45_f45)
• 4x5_gx3v7 (SHORTNAME: f45_g37)
• T62_gx3v7 (SHORTNAME: T62_g37)
• T62_tx0.1v2 (SHORTNAME: T62_t12)
• T62_gx1v6 (SHORTNAME: T62_g16)
• T31_T31 (SHORTNAME: T31_T31)
• T31_gx3v7 (SHORTNAME: T31_g37)
• T42_T42 (SHORTNAME: T42_T42)
• 10x15_10x15 (SHORTNAME: f10_f10)
• ne30np4_1.9x2.5_gx1v6 (SHORTNAME: ne30_f19_g16)

Initializing the Model Setup:

The initial setup of the model on TCS is simplified with the short script below

#!/bin/bash

export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/$USER
export MACH=tcs
export COMPSET=B_1850_CN
export RES=f19_g16
export CASEROOT=~/runs/ccsm4_comp-${COMPSET}_res-${RES}

cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES 

This script creates an 1850 control with all components of the model fully active and carbon nitrogen cycling in the land component, The resolution is 1.9x2.5 in the atmosphere and x1 in the ocean. The file is created in the ~/run directory:

For valid component sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/a2967.html For information on resolution sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/x42.html#ccsm_grids

Load Balancing:

For the NCAR bluefire load balancing table for a select set of simulations see http://www.cesm.ucar.edu/models/cesm1.0/timing/

cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16

edit env_mach_pes.xml

<entry id="NTASKS_ATM"   value="448"  />   
<entry id="NTHRDS_ATM"   value="1"  />   
<entry id="ROOTPE_ATM"   value="0"  />   

<entry id="NTASKS_LND"   value="320"  />   
<entry id="NTHRDS_LND"   value="1"  />   
<entry id="ROOTPE_LND"   value="160"  />   

<entry id="NTASKS_ICE"   value="64"  />   
<entry id="NTHRDS_ICE"   value="1"  />   
<entry id="ROOTPE_ICE"   value="0"  />   

<entry id="NTASKS_OCN"   value="256"  />   
<entry id="NTHRDS_OCN"   value="1"  />   
<entry id="ROOTPE_OCN"   value="224"  />   

<entry id="NTASKS_CPL"   value="224"  />   
<entry id="NTHRDS_CPL"   value="1"  />   
<entry id="ROOTPE_CPL"   value="0"  />   

<entry id="NTASKS_GLC"   value="1"  />   
<entry id="NTHRDS_GLC"   value="1"  />   
<entry id="ROOTPE_GLC"   value="0"  />   

<entry id="PSTRID_ATM"   value="1"  />   
<entry id="PSTRID_LND"   value="1"  />   
<entry id="PSTRID_ICE"   value="1"  />   
<entry id="PSTRID_OCN"   value="1"  />   
<entry id="PSTRID_CPL"   value="1"  />   
<entry id="PSTRID_GLC"   value="1"  />   

Once this file is modified you can configure the case

./configure -case

You will notice that configure will change the file the you just edited and you can see the total processors used by the simulation (704 or 11 nodes in this case):

<entry id="TOTALPES"   value="704"  />   
<entry id="PES_LEVEL"   value="1r"  />   
<entry id="MAX_TASKS_PER_NODE"   value="64"  />   
<entry id="PES_PER_NODE"   value="64"  />   
<entry id="CCSM_PCOST"   value="-3"  />   
<entry id="CCSM_TCOST"   value="0"  />   
<entry id="CCSM_ESTCOST"   value="-3"  />   

The task geometry used by loadleveler on TCS is located in the file: ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run

Now compile the model with:

./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build

One of the pre-processing steps in this build sequence is to fetch inputdat sets (initial and boundary conditions) from the NCAR SVN server. You may want to do this yourself before you build on the datamover1 node if there is a large amount of initial condition data to transfer from the NCAR repository. datamover1 has a high bandwidth connection to the outside. Note: We have most of the input data on /project/ccsm already so this step will not be required for the more common configurations.

> ssh datamover1
Last login: Wed Jul  7 16:38:14 2010 from tcs-f11n06-gpfs
user@gpc-logindm01:~>cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16
user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>
user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>./check_input_data -inputdata /project/ccsm/inputdata -export
Input Data List Files Found:
./Buildconf/cam.input_data_list
./Buildconf/clm.input_data_list
./Buildconf/cice.input_data_list
./Buildconf/pop2.input_data_list
./Buildconf/cpl.input_data_list
export https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc /project/ccsm/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc ..... success

Setting the Simulation Length:

The amount of time that you would like to run the model can be set by editing env_run.xml at anytime in the setup sequence

<entry id="RESUBMIT"   value="10"  />   

<entry id="STOP_OPTION"   value="nmonths"  />   

<entry id="STOP_N"   value="12"  />   

These settings will tell the model to checkpoint after each model year (12 months) and run for a total of 10 years (10 checkpoints)

Running CCSM4 on the Distributed System (TCS):

The model is now ready to be submitted to the TCS batch queue

llsubmit ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run

Once the model has run through a checkpoint timing information on the simulation will be found in:

~/runs/ccsm4_comp-B_1850_CN_res-f19_g16/timing

Standard output from the model can be followed during runtime by going to:

/scratch/guido/ccsm4_comp-B_1850_CN_res-f19_g16/run

and running

tail -f <component_log_file>

The model will archive the NetCDF output in:

/scratch/$USER/archive

Cloning Simulations

A useful command that allow for the setup of multiple runs quickly is the clone command. It allows for the cloning of a case quickly (so there is no need to run the setup script above every time)

cd ~/runs
/project/ccsm/ccsm4_0_current/scripts/create_clone -clone ccsm4_comp-B_1850_CN_res-f09_g16 -case ccsm4_comp-B_1850_CN_res-f09_g16_clone -v

To change the load balancing (env_mach_pes.xml) in a current simulation setup or other parameters you can do a clean build to make sure the model is rebuilt properly:

./configure -cleanmach
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.clean_build
./configure -case
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build

Running CCSM4 on GPC

The setup script is almost identical:

#!/bin/bash

export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/guido
export MACH=gpc
export COMPSET=B_1850_CN
export RES=f09_g16
export CASEROOT=~/runs/ccsm4gpc_comp-${COMPSET}_res-${RES}

cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES 

To load balance and run the model follow the steps above: The env_mach_pes.xml configuration files needs to be modified as follows:

<entry id="MAX_TASKS_PER_NODE"   value="8"  />   
<entry id="PES_PER_NODE"   value="8"  />   

Use qsub to submit the model to the GPC cluster:

qsub ccsm4gpc_comp-B_1850_CN_res-f19_g16.tcs.run