Running CCSM4
|
WARNING: SciNet is in the process of replacing this wiki with a new documentation site. For current information, please go to https://docs.scinet.utoronto.ca |
It is important to point out that all updates to the model system will only occur with CESM1.0 updates, not with CCSM4.0. It is also important to note that CCSM4 is a subset of CESM1. Although CESM1 supersedes CCSM4, users can run all CCSM4 experiments from the CESM1 code base.
Note: To run interactively on GPC interactive queue or TCS on small subset of interactive processors, see the instructions below.
The scientifically validated CESM1 runs are found in the list below (including a complete list of the model resolutions):
/project/ccsm/cesm1_current/scripts/create_newcase --list
CESM1.0 README For both a quick start as well as a detailed summary of creating and running a CESM model case, see the CESM1.0 User's Guide at http://www.cesm.ucar.edu/models/cesm1.0 IMPORTANT INFORMATION ABOUT SCIENTIFIC VALIDATION CESM1.0 has the flexibility to configure cases with many different combinations of component models, grids, and model settings, but this version of CESM has only been validated scientifically for the following fully active configurations: 1.9x2.5_gx1v6 B_1850_CN 1.9x2.5_gx1v6 B_1850_RAMPCO2_CN 1.9x2.5_gx1v6 B_1850-2000_CN 1.9x2.5_gx1v6 B_1850_CAM5 0.9x1.25_gx1v6 B_1850_CN 0.9x1.25_gx1v6 B_1850_RAMPCO2_CN 0.9x1.25_gx1v6 B_1850-2000_CN 0.9x1.25_gx1v6 B_1850_BGC-BPRP 0.9x1.25_gx1v6 B_1850_BGC-BDRD 0.9x1.25_gx1v6 B_1850-2000_BGC-BPRP 0.9x1.25_gx1v6 B_1850-2000_BGC-BDRD 0.9x1.25_gx1v6 B_1850_CN_CHEM 0.9x1.25_gx1v6 B_1850-2000_CN_CHEM 1.9x2.5_gx1v6 B_1850_WACCM_CN 1.9x2.5_gx1v6 B_1850-2000_WACCM_CN T31_gx3v7 B_1850_CN If the user is interested in running a "stand-alone" component configuration, the following model configurations have been validated scientifically and have associated diagnostic output as part of the release: 1.9x2.5_1.9x2.5 F_2000_WACCM 1.9x2.5_1.9x2.5 F_AMIP_CAM5 1.9x2.5_1.9x2.5 F_AMIP_CN 0.9x1.25_0.9x1.25 F_AMIP_CN 0.9x1.25_gx1v6 I_2000 0.9x1.25_gx1v6 I_2000_CN T62_gx1v6 C_NORMAL_YEAR For more information regarding alternative component configurations, please refer to the individual component web pages at http://www.cesm.ucar.edu/models/cesm1.0
CESM1 RESOLUTIONS: name (shortname)
pt1_pt1 (pt1)
0.23x0.31_0.23x0.31 (f02_f02)
0.23x0.31_gx1v6 (f02_g16)
0.23x0.31_tx0.1v2 (f02_t12)
0.47x0.63_0.47x0.63 (f05_f05)
0.47x0.63_gx1v6 (f05_g16)
0.47x0.63_tx0.1v2 (f05_t12)
0.9x1.25_0.9x1.25 (f09_f09)
0.9x1.25_gx1v6 (f09_g16)
1.9x2.5_1.9x2.5 (f19_f19)
1.9x2.5_gx1v6 (f19_g16)
4x5_4x5 (f45_f45)
4x5_gx3v7 (f45_g37)
T62_gx3v7 (T62_g37)
T62_tx0.1v2 (T62_t12)
T62_gx1v6 (T62_g16)
T31_T31 (T31_T31)
T31_gx3v7 (T31_g37)
T42_T42 (T42_T42)
10x15_10x15 (f10_f10)
ne30np4_1.9x2.5_gx1v6 (ne30_f19_g16)
ne240np4_0.23x0.31_gx1v6 (ne240_f02_g16)
T85_T85 (T85_T85)
COMPSETS: name (shortname): description
A_PRESENT_DAY (A)
Description: All data model
A_GLC (AG)
Description: All data model plus glc (glacier model)
B_2000 (B)
Description: All active components, present day
B_2000_CN (BCN)
Description: all active components, present day, with CN (Carbon Nitrogen) in clm
B_1850_CAM5 (B1850C5)
Description: All active components, pre-industrial, cam5 physics
B_1850 (B1850)
Description: All active components, pre-industrial
B_1850_CN (B1850CN)
Description: all active components, pre-industrial, with CN (Carbon Nitrogen) in CLM
B_2000_CN_CHEM (B2000CNCHM)
Description: All active components, pre-industrial, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm
B_1850_CN_CHEM (B1850CNCHM)
Description: All active components, pre-industrial, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm
B_1850_RAMPCO2_CN (B1850RMCN)
Description: All active components, pre-industirial with co2 ramp, with CN (Carbon Nitrogen) in CLM
B_1850-2000 (B20TR)
Description: All active components, 1850 to 2000 transient
B_1850-2000_CN (B20TRCN)
Description: All active components, 1850 to 2000 transient, with CN (Carbon Nitrogen) in CLM
B_1850-2000_CN_CHEM (B20TRCNCHM)
Description: All active components, 1850 to 2000 transient, with CN (Carbon Nitrogen) in CLM and super_fast_llnl chem in atm
B_1850-2000_CAM5 (B20TRC5)
Description: All active components, 1850 to 2000 transient, cam5 physics
B_2000_GLC (BG)
Description: all active components, with active glc
B_2000_TROP_MOZART (BMOZ)
Description: All active components, with trop_mozart
B_1850_WACCM (B1850W)
Description: all active components, pre-industrial, with waccm
B_1850_WACCM_CN (B1850WCN)
Description: all active components, pre-industrial, with waccm and CN
B_1850-2000_WACCM_CN (B20TRWCN)
Description: All active components, 1850 to 2000 transient, WACCM with CN (Carbon Nitrogen) in CLM
B_1955-2005_WACCM_CN (B55TRWCN)
Description: All active components, 1955 to 2000 transient, WACCM with daily solar data and SPEs, CLM with CN
B_1850_BGC-BPRP (B1850BPRP)
Description: All active components, pre-industrial, CN in CLM, ECO in POP, BGC CO2=prog, rad CO2=prog
B_1850_BGC-BDRD (B1850BDRD)
Description: All active components, pre-industrial, CN in CLM, ECO in POP, BGC CO2=diag, rad CO2=diag
B_1850-2000_BGC-BPRP (B20TRBPRP)
Description: All active components, 1850 to 2000 transient, CN in CLM, ECO in POP, BGC CO2=prog, rad CO2=prog
B_1850-2000_BGC-BDRD (B20TRBDRD)
Description: All active components, 1850 to 2000 transient, CN in CLM, ECO in POP, BGC CO2=diag, rad CO2=diag
C_NORMAL_YEAR_ECOSYS (CECO)
Description: Active ocean model with ecosys and with COREv2 normal year forcing
C_NORMAL_YEAR (C)
Description: Active ocean model with COREv2 normal year forcing
D_NORMAL_YEAR (D)
Description: Active ice model with COREv2 normal year forcing
E_2000 (E)
Description: Fully active cam and ice with som ocean, present day
E_2000_GLC (EG)
Description: Fully active cam and ice with som ocean and glc, present day
E_1850_CN (E1850CN)
Description: Pre-industrial fully active ice and som ocean, with CN
E_1850_CAM5 (E1850C5)
Description: Pre-industrial fully active ice and som ocean, cam5 physics
F_AMIP_CN (FAMIPCN)
Description: AMIP run for CMIP5 protocol - valid only for 1 degree cam/clm/pres-cice
F_AMIP_CAM5 (FAMIPC5)
Description: AMIP run for CMIP5 protocol with cam5
F_1850 (F1850)
Description: Pre-industrial cam/clm with prescribed ice/ocn
F_1850_CAM5 (F1850C5)
Description: Pre-industrial cam/clm with prescribed ice/ocn, cam5 physics
F_2000 (F)
Description: Stand-alone cam default, prescribed ocn/ice
F_2000_CAM5 (FC5)
Description: Stand-alone cam default, prescribed ocn/ice, cam5 physics
F_2000_CN (FCN)
Description: Stand-alone cam default, prescribed ocn/ice with CN
F_1850-2000_CN (F20TRCN)
Description: 20th Century transient stand-alone cam default, prescribed ocn/ice, with CN
F_2000_GLC (FG)
Description: Stand-alone cam default, prescribed ocn/ice, glc (glacier model)
F_1850_CN_CHEM (F1850CNCHM)
Description: stand-alone cam/clm, pre-industrial, with CN in CLM, super_fast_llnl chem in cam
F_1850_WACCM (F1850W)
Description: Pre-industrial cam/clm with prescribed ice/ocn
F_2000_WACCM (FW)
Description: present-day cam/clm with prescribed ice/ocn
G_1850_ECOSYS (G1850ECO)
Description: 1850 control for pop-ecosystem/cice/datm7/dlnd-rx1
G_NORMAL_YEAR (G)
Description: Coupled ocean ice with COREv2 normal year forcing
H_PRESENT_DAY (H)
Description: Coupled ocean ice slnd
I_2000 (I)
Description: Active land model with QIAN atm input data for 2003 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000
I_1850 (I1850)
Description: Active land model with QIAN atm input data for 1948 to 1972 and Satellite phenology (SP), CO2 level and Aerosol deposition for 1850
I_2000_GLC (IG)
Description: Active glacier model and active land model with QIAN atm input data for 2003 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000
I_1948-2004 (I4804)
Description: Active land model with QIAN atm input data for 1948 to 2004 and Satellite phenology (SP), CO2 level and Aerosol deposition for 2000
I_1850-2000 (I8520)
Description: Active land model with QIAN atm input data for 1948 to 2004 and transient Satellite phenology (SP), and Aerosol deposition from 1850 to 2000 and 2000 CO2 level
I_2000_CN (ICN)
Description: Active land model with QIAN atm input data for 2003 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 2000
I_1850_CN (I1850CN)
Description: Active land model with QIAN atm input data for 1948 to 1972 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 1850
I_1948-2004_CN (I4804CN)
Description: Active land model with QIAN atm input data for 1948 to 2004 and CN (Carbon Nitrogen) biogeochemistry, CO2 level and Aerosol deposition for 2000
I_1850-2000_CN (I8520CN)
Description: Active land model with QIAN atm input data for 1948 to 1972 and transient CN, Aerosol dep from 1850 to 2000 and 2000 CO2 level
S_PRESENT_DAY (S)
Description: All stub models plus xatm
X_PRESENT_DAY (X)
Description: All dead model
XG_PRESENT_DAY (XG)
Description: All dead model and cism
MACHINES: name (description)
tcs (U of T IBM p6, os is AIX, 32 pes/node, batch system is Moab/LoadLeveler)
gpc (U of T iDataPlex intel cluster, os is linux, 8 pes/node, batch system is Moab/Torque)
bluefire (NCAR IBM p6, os is AIX, 32 pes/node, batch system is LSF)
brutus_po (Brutus Linux Cluster ETH (pgi/9.0-1 with open_mpi/1.4.1), 16 pes/node, batch system LSF, added by UB)
brutus_pm (Brutus Linux Cluster ETH (pgi/9.0-1 with mvapich2/1.4rc2), 16 pes/node, batch system LSF, added by UB)
brutus_io (Brutus Linux Cluster ETH (intel/10.1.018 with open_mpi/1.4.1), 16 pes/node, batch system LSF, added by UB)
brutus_im (Brutus Linux Cluster ETH (intel/10.1.018 with mvapich2/1.4rc2), 16 pes/node, batch system LSF, added by UB)
edinburgh_lahey (NCAR CGD Linux Cluster (lahey), 8 pes/node, batch system is PBS)
edinburgh_pgi (NCAR CGD Linux Cluster (pgi), 8 pes/node, batch system is PBS)
edinburgh_intel (NCAR CGD Linux Cluster (intel), 8 pes/node, batch system is PBS)
franklin (NERSC XT4, os is CNL, 4 pes/node, batch system is PBS)
hadley (UCB Linux Cluster, os is Linux (ia64), batch system is PBS)
hopper (NERSC XT5, os is CNL, 8 pes/node, batch system is PBS)
intrepid (ANL IBM BG/P, os is BGP, 4 pes/node, batch system is cobalt)
jaguar (ORNL XT4, os is CNL, 4 pes/node, batch system is PBS)
jaguarpf (ORNL XT5, os is CNL, 12 pes/node, batch system is PBS)
kraken (NICS/UT/teragrid XT5, os is CNL, 12 pes/node)
lynx_pgi (NCAR XT5, os is CNL, 12 pes/node, batch system is PBS)
midnight (ARSC Sun Cluster, os is Linux (pgi), batch system is PBS)
pleiades (NASA/AMES Linux Cluster, Linux (ia64), Altix ICE, 3.0 GHz Harpertown processors, 8 pes/node and 8 GB of memory, batch system is PBS)
pleiades_wes (NASA/AMES Linux Cluster, Linux (ia64), Altix ICE, 2.93 GHz Westmere processors, 12 pes/node and 24 GB of memory, batch system is PBS)
prototype_atlas (LLNL Linux Cluster, Linux (pgi), 8 pes/node, batch system is Moab)
prototype_hera (LLNL Linux Cluster, Linux (pgi), 16 pes/node, batch system is Moab)
prototype_columbia (NASA Ames Linux Cluster, Linux (ia64), 2 pes/node, batch system is PBS)
prototype_frost (NCAR IBM BG/L, os is BGL, 8 pes/node, batch system is cobalt)
prototype_nyblue (SUNY IBM BG/L, os is BGL, 8 pes/node, batch system is cobalt)
prototype_ranger (TACC Linux Cluster, Linux (pgi), 1 pes/node, batch system is SGE)
prototype_ubgl (LLNL IBM BG/L, os is BGL, 2 pes/node, batch system is Moab)
generic_ibm (generic ibm power system, os is AIX, batch system is LoadLeveler, user-defined)
generic_xt (generic CRAY XT, os is CNL, batch system is PBS, user-defined)
generic_linux_pgi (generic linux (pgi), os is Linux, batch system is PBS, user-defined)
generic_linux_lahey (generic linux (lahey), os is Linux, batch system is PBS, user-defined)
generic_linux_intel (generic linux (intel), os is Linux, batch system is PBS, user-defined)
generic_linux_pathscale (generic linux (pathscale), os is Linux, batch system is PBS, user-defined)
Initializing the Model Setup:
The initial setup of the model on TCS is simplified with the short script below
#!/bin/bash
export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/$USER
export MACH=tcs
export COMPSET=B_1850_CN
export RES=f19_g16
export CASEROOT=~/runs/ccsm4_comp-${COMPSET}_res-${RES}
cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES -scratchroot $SCRATCH
NOTE: $CCSMROOT should point to the model code version in /project/ccsm with the "_current" after it. The same for CESM1
NOTE: $CASEROOT which will specify your model run naming conventions will not archive properly (to short term archiving directory i.e. /scratch) if your CASEROOT name is too long. It is best to keep it short.
This script creates an 1850 control with all components of the model fully active and carbon nitrogen cycling in the land component, The resolution is 1.9x2.5 in the atmosphere and x1 in the ocean. The file is created in the ~/run directory:
For valid component sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/a2967.html For information on resolution sets see: http://www.cesm.ucar.edu/models/ccsm4.0/ccsm_doc/x42.html#ccsm_grids
Load Balancing:
For the NCAR bluefire load balancing table for a select set of simulations see: CESM1: http://www.cesm.ucar.edu/models/cesm1.0/timing/ CCSM4: http://www.cesm.ucar.edu/models/ccsm4.0/timing/
cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16
edit env_mach_pes.xml
<entry id="NTASKS_ATM" value="448" /> <entry id="NTHRDS_ATM" value="1" /> <entry id="ROOTPE_ATM" value="0" /> <entry id="NTASKS_LND" value="320" /> <entry id="NTHRDS_LND" value="1" /> <entry id="ROOTPE_LND" value="160" /> <entry id="NTASKS_ICE" value="64" /> <entry id="NTHRDS_ICE" value="1" /> <entry id="ROOTPE_ICE" value="0" /> <entry id="NTASKS_OCN" value="256" /> <entry id="NTHRDS_OCN" value="1" /> <entry id="ROOTPE_OCN" value="224" /> <entry id="NTASKS_CPL" value="224" /> <entry id="NTHRDS_CPL" value="1" /> <entry id="ROOTPE_CPL" value="0" /> <entry id="NTASKS_GLC" value="1" /> <entry id="NTHRDS_GLC" value="1" /> <entry id="ROOTPE_GLC" value="0" /> <entry id="PSTRID_ATM" value="1" /> <entry id="PSTRID_LND" value="1" /> <entry id="PSTRID_ICE" value="1" /> <entry id="PSTRID_OCN" value="1" /> <entry id="PSTRID_CPL" value="1" /> <entry id="PSTRID_GLC" value="1" />
Once this file is modified you can configure the case
./configure -case
You will notice that configure will change the file the you just edited and you can see the total processors used by the simulation (704 or 11 nodes in this case):
<entry id="TOTALPES" value="704" /> <entry id="PES_LEVEL" value="1r" /> <entry id="MAX_TASKS_PER_NODE" value="64" /> <entry id="PES_PER_NODE" value="64" /> <entry id="CCSM_PCOST" value="-3" /> <entry id="CCSM_TCOST" value="0" /> <entry id="CCSM_ESTCOST" value="-3" />
Note: Rather than modifying the load balancing manually, NCAR has written a script that resides in your $CASE running directory that allows you to modify the individual component CPU allocation without playing with the env_mach_pes.xml file:
To try a different configuration we might want 8 cpus running the OCN component continually and the remaining 24 cpus running atm on 24 then LND, ICE and CPL on 8 each. To set this up you enter;
configure -cleanmach xmlchange -file env_mach_pes.xml -id NTASKS_ATM -val 24 xmlchange -file env_mach_pes.xml -id NTASKS_LND -val 8 xmlchange -file env_mach_pes.xml -id NTASKS_ICE -val 8 xmlchange -file env_mach_pes.xml -id ROOTPE_ICE -val 8 xmlchange -file env_mach_pes.xml -id NTASKS_CPL -val 8 xmlchange -file env_mach_pes.xml -id ROOTPE_CPL -val 16 xmlchange -file env_mach_pes.xml -id NTASKS_OCN -val 8 xmlchange -file env_mach_pes.xml -id ROOTPE_OCN -val 24 configure -case
Then build and resubmit
The task geometry used by loadleveler on TCS is located in the file: ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run
Ensure that the proper modules are loaded:
Currently Loaded Modulefiles:
1) ncl/5.1.1 3) netcdf/4.0.1_nc3 5) xlf/13.1 2) nco/3.9.6 4) parallel-netcdf/1.1.1 6) vacpp/11.1
Now compile the model with:
./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build
One of the pre-processing steps in this build sequence is to fetch inputdat sets (initial and boundary conditions) from the NCAR SVN server. You may want to do this yourself before you build on the datamover1 node if there is a large amount of initial condition data to transfer from the NCAR repository. datamover1 has a high bandwidth connection to the outside. Note: We have most of the input data on /project/ccsm already so this step will not be required for the more common configurations.
> ssh datamover1 Last login: Wed Jul 7 16:38:14 2010 from tcs-f11n06-gpfs user@gpc-logindm01:~>cd ~/runs/ccsm4_comp-B_1850_CN_res-f19_g16 user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16> user@gpc-logindm01:~/runs/ccsm4_comp-B_1850_CN_res-f19_g16>./check_input_data -inputdata /project/ccsm/inputdata -export Input Data List Files Found: ./Buildconf/cam.input_data_list ./Buildconf/clm.input_data_list ./Buildconf/cice.input_data_list ./Buildconf/pop2.input_data_list ./Buildconf/cpl.input_data_list export https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc /project/ccsm/inputdata/atm/cam/chem/trop_mozart_aero/aero/aero_1.9x2.5_L26_1850clim_c091112.nc ..... success
Setting the Simulation Length:
The amount of time that you would like to run the model can be set by editing env_run.xml at anytime in the setup sequence
<entry id="RESUBMIT" value="10" /> <entry id="STOP_OPTION" value="nmonths" /> <entry id="STOP_N" value="12" />
These settings will tell the model to checkpoint after each model year (12 months) and run for a total of 10 years (10 checkpoints)
Running CCSM4 on the Distributed System (TCS):
The model is now ready to be submitted to the TCS batch queue
llsubmit ccsm4_comp-B_1850_CN_res-f19_g16.tcs.run
Once the model has run through a checkpoint timing information on the simulation will be found in:
~/runs/ccsm4_comp-B_1850_CN_res-f19_g16/timing
Standard output from the model can be followed during runtime by going to:
/scratch/guido/ccsm4_comp-B_1850_CN_res-f19_g16/run
and running
tail -f <component_log_file>
The model will archive the NetCDF output in:
/scratch/$USER/archive
Cloning Simulations
A useful command that allow for the setup of multiple runs quickly is the clone command. It allows for the cloning of a case quickly (so there is no need to run the setup script above every time)
cd ~/runs /project/ccsm/ccsm4_0_current/scripts/create_clone -clone ccsm4_comp-B_1850_CN_res-f09_g16 -case ccsm4_comp-B_1850_CN_res-f09_g16_clone -v
To change the load balancing (env_mach_pes.xml) in a current simulation setup or other parameters you can do a clean build to make sure the model is rebuilt properly:
./configure -cleanmach ./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.clean_build ./configure -case ./ccsm4_comp-B_1850_CN_res-f19_g16.tcs.build
Running CCSM4 on GPC
The setup script is almost identical:
#!/bin/bash
export CCSMROOT=/project/ccsm/ccsm4_0_current
export SCRATCH=/scratch/guido
export MACH=gpc
export COMPSET=B_1850_CN
export RES=f09_g16
export CASEROOT=~/runs/ccsm4gpc_comp-${COMPSET}_res-${RES}
cd $CCSMROOT/scripts
./create_newcase -verbose -case $CASEROOT -mach $MACH -compset $COMPSET -res $RES
To load balance and run the model follow the steps above: The env_mach_pes.xml configuration files needs to be modified as follows:
<entry id="MAX_TASKS_PER_NODE" value="8" /> <entry id="PES_PER_NODE" value="8" />
Use qsub to submit the model to the GPC cluster:
qsub ccsm4gpc_comp-B_1850_CN_res-f19_g16.tcs.run
Running CCSM4 interactively on TCS
In the run script you need to specify the following set and unset of environment variables. You also need a hostfile that will describe the node procs that you will be using
unsetenv MP_EUILIB setenv MP_PROCS 16 setenv MP_NODES 1 /usr/bin/poe /project/ccsm/bin/ccsm_launch ./ccsm.exe -hfile /project/<user>/runs/<casename>/hostfile
and the hostfile looks like (for 16 threads of tcs02)
--> more hostfile tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06 tcs-f11n06