Running CCSM4
The scientifically validated CCSM4 runs are found in the list below (including a complete list of the model resolutions):
CCSM has significant flexibility to configure cases with respect to components, grids, and model settings. This version of CCSM has only be validated scientifically for the following full active configurations:
- 1.9x2.5_gx1v6 B_1850_CN
- 1.9x2.5_gx1v6 B_1850_RAMPCO2_CN
- 0.9x1.25_gx1v6 B_1850_CN
- 0.9x1.25_gx1v6 B_1850_RAMPCO2_CN
If the user is interested in running a "stand-alone" component configuration, the following model configurations have been validated scientifically and have associated diagnostic output as part of the release.
- 0.9x1.25_gx1v6 F_AMIP_1DEG
- 1.9x2.5_gx1v6 F_AMIP_2DEG
- 0.9x1.25_gx1v6 I and ICN
- T62_gx1v6 C
Please refer to the individual component release web pages above for information regarding alternative component configurations.
- pt1_pt1 (SHORTNAME: pt1)
- 0.47x0.63_0.47x0.63 (SHORTNAME: f05_f05)
- 0.47x0.63_gx1v6 (SHORTNAME: f05_g16)
- 0.47x0.63_tx0.1v2 (SHORTNAME: f05_t12)
- 0.9x1.25_0.9x1.25 (SHORTNAME: f09_f09)
- 0.9x1.25_gx1v6 (SHORTNAME: f09_g16)
- 1.9x2.5_1.9x2.5 (SHORTNAME: f19_f19)
- 1.9x2.5_gx1v6 (SHORTNAME: f19_g16)
- 4x5_4x5 (SHORTNAME: f45_f45)
- 4x5_gx3v7 (SHORTNAME: f45_g37)
- T62_gx3v7 (SHORTNAME: T62_g37)
- T62_tx0.1v2 (SHORTNAME: T62_t12)
- T62_gx1v6 (SHORTNAME: T62_g16)
- T31_T31 (SHORTNAME: T31_T31)
- T31_gx3v7 (SHORTNAME: T31_g37)
- T42_T42 (SHORTNAME: T42_T42)
- 10x15_10x15 (SHORTNAME: f10_f10)
- ne30np4_1.9x2.5_gx1v6 (SHORTNAME: ne30_f19_g16)
Initializing the Model Setup:
The initial setup of the model on TCS is simplified with the short script below
#!/bin/bash
export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/$USER
export MACH=tcs
export COMPSET=B_1850_CN
export RES=f19_g16
export CASEROOT=~/runs/ccsm4_comp-${COMPSET}_res-${RES}
cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES
This script creates an 1850 control with all components of the model fully active and carbon nitrogen cycling in the land component, The resolution is 1.9x2.5 in the atmosphere and x1 in the ocean. The file is created in the ~/run directory:
For valid component sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/a2967.html For information on resolution sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/x42.html#ccsm_grids
Load Balancing:
For the NCAR bluefire load balancing table for a select set of simulations see http://www.cesm.ucar.edu/models/cesm1.0/timing/
cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16
edit env_mach_pes.xml
<entry id="NTASKS_ATM" value="448" /> <entry id="NTHRDS_ATM" value="1" /> <entry id="ROOTPE_ATM" value="0" /> <entry id="NTASKS_LND" value="320" /> <entry id="NTHRDS_LND" value="1" /> <entry id="ROOTPE_LND" value="160" /> <entry id="NTASKS_ICE" value="64" /> <entry id="NTHRDS_ICE" value="1" /> <entry id="ROOTPE_ICE" value="0" /> <entry id="NTASKS_OCN" value="256" /> <entry id="NTHRDS_OCN" value="1" /> <entry id="ROOTPE_OCN" value="224" /> <entry id="NTASKS_CPL" value="224" /> <entry id="NTHRDS_CPL" value="1" /> <entry id="ROOTPE_CPL" value="0" /> <entry id="NTASKS_GLC" value="1" /> <entry id="NTHRDS_GLC" value="1" /> <entry id="ROOTPE_GLC" value="0" /> <entry id="PSTRID_ATM" value="1" /> <entry id="PSTRID_LND" value="1" /> <entry id="PSTRID_ICE" value="1" /> <entry id="PSTRID_OCN" value="1" /> <entry id="PSTRID_CPL" value="1" /> <entry id="PSTRID_GLC" value="1" />
Once this file is modified you can configure the case
./configure -case
You will notice that configure will change the file the you just edited and you can see the total processors used by the simulation (704 or 11 nodes in this case):
<entry id="TOTALPES" value="704" /> <entry id="PES_LEVEL" value="1r" /> <entry id="MAX_TASKS_PER_NODE" value="64" /> <entry id="PES_PER_NODE" value="64" /> <entry id="CCSM_PCOST" value="-3" /> <entry id="CCSM_TCOST" value="0" /> <entry id="CCSM_ESTCOST" value="-3" />
Note: Rather than modifying the load balancing manually, NCAR has written a script that resides in your $CASE running directory that allows you to modify the individual component CPU allocation without playing with the env_mach_pes.xml file:
To try a different configuration we might want 8 cpus running the OCN component continually and the remaining 24 cpus running atm on 24 then LND, ICE and CPL on 8 each. To set this up you enter;
configure -cleanmach xmlchange -file env_mach_pes.xml -id NTASKS_ATM -val 24 xmlchange -file env_mach_pes.xml -id NTASKS_LND -val 8 xmlchange -file env_mach_pes.xml -id NTASKS_ICE -val 8 xmlchange -file env_mach_pes.xml -id ROOTPE_ICE -val 8 xmlchange -file env_mach_pes.xml -id NTASKS_CPL -val 8 xmlchange -file env_mach_pes.xml -id ROOTPE_CPL -val 16 xmlchange -file env_mach_pes.xml -id NTASKS_OCN -val 8 xmlchange -file env_mach_pes.xml -id ROOTPE_OCN -val 24 configure -case
Then build and resubmit
The task geometry used by loadleveler on TCS is located in the file: ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run
Now compile the model with:
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build
One of the pre-processing steps in this build sequence is to fetch inputdat sets (initial and boundary conditions) from the NCAR SVN server. You may want to do this yourself before you build on the datamover1 node if there is a large amount of initial condition data to transfer from the NCAR repository. datamover1 has a high bandwidth connection to the outside. Note: We have most of the input data on /project/ccsm already so this step will not be required for the more common configurations.
> ssh datamover1 Last login: Wed Jul 7 16:38:14 2010 from tcs-f11n06-gpfs user@gpc-logindm01:~>cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16 user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16> user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>./check_input_data -inputdata /project/ccsm/inputdata -export Input Data List Files Found: ./Buildconf/cam.input_data_list ./Buildconf/clm.input_data_list ./Buildconf/cice.input_data_list ./Buildconf/pop2.input_data_list ./Buildconf/cpl.input_data_list export https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc /project/ccsm/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc ..... success
Setting the Simulation Length:
The amount of time that you would like to run the model can be set by editing env_run.xml at anytime in the setup sequence
<entry id="RESUBMIT" value="10" /> <entry id="STOP_OPTION" value="nmonths" /> <entry id="STOP_N" value="12" />
These settings will tell the model to checkpoint after each model year (12 months) and run for a total of 10 years (10 checkpoints)
Running CCSM4 on the Distributed System (TCS):
The model is now ready to be submitted to the TCS batch queue
llsubmit ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run
Once the model has run through a checkpoint timing information on the simulation will be found in:
~/runs/ccsm4_comp-B_1850_CN_res-f19_g16/timing
Standard output from the model can be followed during runtime by going to:
/scratch/guido/ccsm4_comp-B_1850_CN_res-f19_g16/run
and running
tail -f <component_log_file>
The model will archive the NetCDF output in:
/scratch/$USER/archive
Cloning Simulations
A useful command that allow for the setup of multiple runs quickly is the clone command. It allows for the cloning of a case quickly (so there is no need to run the setup script above every time)
cd ~/runs /project/ccsm/ccsm4_0_current/scripts/create_clone -clone ccsm4_comp-B_1850_CN_res-f09_g16 -case ccsm4_comp-B_1850_CN_res-f09_g16_clone -v
To change the load balancing (env_mach_pes.xml) in a current simulation setup or other parameters you can do a clean build to make sure the model is rebuilt properly:
./configure -cleanmach ./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.clean_build ./configure -case ./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build
Running CCSM4 on GPC
The setup script is almost identical:
#!/bin/bash
export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/guido
export MACH=gpc
export COMPSET=B_1850_CN
export RES=f09_g16
export CASEROOT=~/runs/ccsm4gpc_comp-${COMPSET}_res-${RES}
cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES
To load balance and run the model follow the steps above: The env_mach_pes.xml configuration files needs to be modified as follows:
<entry id="MAX_TASKS_PER_NODE" value="8" /> <entry id="PES_PER_NODE" value="8" />
Use qsub to submit the model to the GPC cluster:
qsub ccsm4gpc_comp-B_1850_CN_res-f19_g16.tcs.run