Running CCSM4

It is important to point out that all updates to the model system will only occur with CESM1.0 updates, not with CCSM4.0. It is also important to note that CCSM4 is a subset of CESM1. Although CESM1 supersedes CCSM4, users can run all CCSM4 experiments from the CESM1 code base.

The scientifically validated CESM1 runs are found in the list below (including a complete list of the model resolutions):

/project/ccsm/cesm1_current/scripts/create_newcase --list

---

                           CESM1.0 README

For both a quick start as well as a detailed summary of creating and running a CESM model case, see the CESM1.0 User's Guide at http://www.cesm.ucar.edu/models/cesm1.0

IMPORTANT INFORMATION ABOUT SCIENTIFIC VALIDATION

  CESM1.0 has the flexibility to configure cases with many different 
  combinations of component models, grids, and model settings, but this 
  version of CESM has only been validated scientifically for the following 
  fully active configurations:

     1.9x2.5_gx1v6   B_1850_CN
     1.9x2.5_gx1v6   B_1850_RAMPCO2_CN
     1.9x2.5_gx1v6   B_1850-2000_CN

     1.9x2.5_gx1v6   B_1850_CAM5

     0.9x1.25_gx1v6  B_1850_CN
     0.9x1.25_gx1v6  B_1850_RAMPCO2_CN
     0.9x1.25_gx1v6  B_1850-2000_CN

     0.9x1.25_gx1v6  B_1850_BGC-BPRP
     0.9x1.25_gx1v6  B_1850_BGC-BDRD
     0.9x1.25_gx1v6  B_1850-2000_BGC-BPRP
     0.9x1.25_gx1v6  B_1850-2000_BGC-BDRD

     0.9x1.25_gx1v6  B_1850_CN_CHEM 
     0.9x1.25_gx1v6  B_1850-2000_CN_CHEM

     1.9x2.5_gx1v6   B_1850_WACCM_CN
     1.9x2.5_gx1v6   B_1850-2000_WACCM_CN
 
     T31_gx3v7       B_1850_CN

  If the user is interested in running a "stand-alone" component configuration, 
  the following model configurations have been validated scientifically and 
  have associated diagnostic output as part of the release:

     1.9x2.5_1.9x2.5    F_2000_WACCM
     1.9x2.5_1.9x2.5    F_AMIP_CAM5
     1.9x2.5_1.9x2.5    F_AMIP_CN
     0.9x1.25_0.9x1.25  F_AMIP_CN

     0.9x1.25_gx1v6  I_2000
     0.9x1.25_gx1v6  I_2000_CN

     T62_gx1v6       C_NORMAL_YEAR

  For more information regarding alternative component configurations, 
  please refer to the individual component web pages at
  http://www.cesm.ucar.edu/models/cesm1.0

---

---

Initializing the Model Setup:

The initial setup of the model on TCS is simplified with the short script below

#!/bin/bash

export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/$USER
export MACH=tcs
export COMPSET=B_1850_CN
export RES=f19_g16
export CASEROOT=~/runs/ccsm4_comp-${COMPSET}_res-${RES}

cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES 

NOTE: CCSMROOT should point to the model code version in /project/ccsm with the "_current" after it. The same for CESM1

This script creates an 1850 control with all components of the model fully active and carbon nitrogen cycling in the land component, The resolution is 1.9x2.5 in the atmosphere and x1 in the ocean. The file is created in the ~/run directory:

For valid component sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/a2967.html For information on resolution sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/x42.html#ccsm_grids

Load Balancing:

For the NCAR bluefire load balancing table for a select set of simulations see: CESM1: http://www.cesm.ucar.edu/models/cesm1.0/timing/ CCSM4: http://www.cesm.ucar.edu/models/ccsm4.0/timing/

cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16

edit env_mach_pes.xml

<entry id="NTASKS_ATM"   value="448"  />   
<entry id="NTHRDS_ATM"   value="1"  />   
<entry id="ROOTPE_ATM"   value="0"  />   

<entry id="NTASKS_LND"   value="320"  />   
<entry id="NTHRDS_LND"   value="1"  />   
<entry id="ROOTPE_LND"   value="160"  />   

<entry id="NTASKS_ICE"   value="64"  />   
<entry id="NTHRDS_ICE"   value="1"  />   
<entry id="ROOTPE_ICE"   value="0"  />   

<entry id="NTASKS_OCN"   value="256"  />   
<entry id="NTHRDS_OCN"   value="1"  />   
<entry id="ROOTPE_OCN"   value="224"  />   

<entry id="NTASKS_CPL"   value="224"  />   
<entry id="NTHRDS_CPL"   value="1"  />   
<entry id="ROOTPE_CPL"   value="0"  />   

<entry id="NTASKS_GLC"   value="1"  />   
<entry id="NTHRDS_GLC"   value="1"  />   
<entry id="ROOTPE_GLC"   value="0"  />   

<entry id="PSTRID_ATM"   value="1"  />   
<entry id="PSTRID_LND"   value="1"  />   
<entry id="PSTRID_ICE"   value="1"  />   
<entry id="PSTRID_OCN"   value="1"  />   
<entry id="PSTRID_CPL"   value="1"  />   
<entry id="PSTRID_GLC"   value="1"  />   

Once this file is modified you can configure the case

./configure -case

You will notice that configure will change the file the you just edited and you can see the total processors used by the simulation (704 or 11 nodes in this case):

<entry id="TOTALPES"   value="704"  />   
<entry id="PES_LEVEL"   value="1r"  />   
<entry id="MAX_TASKS_PER_NODE"   value="64"  />   
<entry id="PES_PER_NODE"   value="64"  />   
<entry id="CCSM_PCOST"   value="-3"  />   
<entry id="CCSM_TCOST"   value="0"  />   
<entry id="CCSM_ESTCOST"   value="-3"  />   

Note: Rather than modifying the load balancing manually, NCAR has written a script that resides in your $CASE running directory that allows you to modify the individual component CPU allocation without playing with the env_mach_pes.xml file:

To try a different configuration we might want 8 cpus running the OCN component continually and the remaining 24 cpus running atm on 24 then LND, ICE and CPL on 8 each. To set this up you enter;

configure -cleanmach
xmlchange -file env_mach_pes.xml -id NTASKS_ATM -val 24
xmlchange -file env_mach_pes.xml -id NTASKS_LND -val 8 
xmlchange -file env_mach_pes.xml -id NTASKS_ICE -val 8
xmlchange -file env_mach_pes.xml -id ROOTPE_ICE -val 8
xmlchange -file env_mach_pes.xml -id NTASKS_CPL -val 8
xmlchange -file env_mach_pes.xml -id ROOTPE_CPL -val 16
xmlchange -file env_mach_pes.xml -id NTASKS_OCN -val 8
xmlchange -file env_mach_pes.xml -id ROOTPE_OCN -val 24
configure -case

Then build and resubmit

The task geometry used by loadleveler on TCS is located in the file: ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run

Ensure that the proper modules are loaded:

Currently Loaded Modulefiles:

 1) ncl/5.1.1               3) netcdf/4.0.1_nc3        5) xlf/13.1
 2) nco/3.9.6               4) parallel-netcdf/1.1.1   6) vacpp/11.1

Now compile the model with:

./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build

One of the pre-processing steps in this build sequence is to fetch inputdat sets (initial and boundary conditions) from the NCAR SVN server. You may want to do this yourself before you build on the datamover1 node if there is a large amount of initial condition data to transfer from the NCAR repository. datamover1 has a high bandwidth connection to the outside. Note: We have most of the input data on /project/ccsm already so this step will not be required for the more common configurations.

> ssh datamover1
Last login: Wed Jul  7 16:38:14 2010 from tcs-f11n06-gpfs
user@gpc-logindm01:~>cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16
user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>
user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>./check_input_data -inputdata /project/ccsm/inputdata -export
Input Data List Files Found:
./Buildconf/cam.input_data_list
./Buildconf/clm.input_data_list
./Buildconf/cice.input_data_list
./Buildconf/pop2.input_data_list
./Buildconf/cpl.input_data_list
export https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc /project/ccsm/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc ..... success

Setting the Simulation Length:

The amount of time that you would like to run the model can be set by editing env_run.xml at anytime in the setup sequence

<entry id="RESUBMIT"   value="10"  />   

<entry id="STOP_OPTION"   value="nmonths"  />   

<entry id="STOP_N"   value="12"  />   

These settings will tell the model to checkpoint after each model year (12 months) and run for a total of 10 years (10 checkpoints)

Running CCSM4 on the Distributed System (TCS):

The model is now ready to be submitted to the TCS batch queue

llsubmit ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run

Once the model has run through a checkpoint timing information on the simulation will be found in:

~/runs/ccsm4_comp-B_1850_CN_res-f19_g16/timing

Standard output from the model can be followed during runtime by going to:

/scratch/guido/ccsm4_comp-B_1850_CN_res-f19_g16/run

and running

tail -f <component_log_file>

The model will archive the NetCDF output in:

/scratch/$USER/archive

Cloning Simulations

A useful command that allow for the setup of multiple runs quickly is the clone command. It allows for the cloning of a case quickly (so there is no need to run the setup script above every time)

cd ~/runs
/project/ccsm/ccsm4_0_current/scripts/create_clone -clone ccsm4_comp-B_1850_CN_res-f09_g16 -case ccsm4_comp-B_1850_CN_res-f09_g16_clone -v

To change the load balancing (env_mach_pes.xml) in a current simulation setup or other parameters you can do a clean build to make sure the model is rebuilt properly:

./configure -cleanmach
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.clean_build
./configure -case
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build

Running CCSM4 on GPC

The setup script is almost identical:

#!/bin/bash

export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/guido
export MACH=gpc
export COMPSET=B_1850_CN
export RES=f09_g16
export CASEROOT=~/runs/ccsm4gpc_comp-${COMPSET}_res-${RES}

cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES 

To load balance and run the model follow the steps above: The env_mach_pes.xml configuration files needs to be modified as follows:

<entry id="MAX_TASKS_PER_NODE"   value="8"  />   
<entry id="PES_PER_NODE"   value="8"  />   

Use qsub to submit the model to the GPC cluster:

qsub ccsm4gpc_comp-B_1850_CN_res-f19_g16.tcs.run