1. www.ccsm.ucar.edu The CCSM2.0 Quick Start Guide Lawrence Buja CCSM Software Engineering Group ccsm@ucar.edu June 29 2002

2. www.ccsm.ucar.edu CCSM2.0 Quick Start Guide What is needed to run CCSM2? Downloading the CCSM2.0 distribution The CCSM source code Input datasets Building the CCSM Running the CCSM CCSM output data

3. www.ccsm.ucar.edu What is needed to run CCSM2? OS: IBM AIX or SGI IRIX64 Compilers: Fortran 90, C Tools: gunzip, gnumake, Perl5, Perl/Tk Permanent & Temporary disk space Libraries: MPI and netCDF Input data : 0.2 GB for T31_gx3 grid, 0.8 GB for T42_gx1v3 grid CCSM2 source code: 35 MB

4. www.ccsm.ucar.edu The CCSM2.0 distribution CCSM2.0 is available via the web from: http://www.cgd.ucar.edu/csm/models/ccsm2.0 ccsm2.0.tar.gz CCSM2.0 source codes ccsm2.0.inputdata.T42_gx1v3.tar input data for T42_gx1v3 resolution ccsm2.0.inputdata.T31_gx3.tar input data for T31_gx3 resolution ccsm2.0.inputdata.T62.tar latm data for T62 resolution ccsm2.0.inputdata.CLM_SPINUP_FILES.tar are CLM datasets gunzip -c ccsm2.0.tar.gz | tar -xf - tar -xf ccsm2.0.inputdata.T42_gx1v3.tar

5. www.ccsm.ucar.edu The CCSM distribution $HOME/ ccsm2.0/ $CSMROOT /fs/$LOGNAME/ inputdata/ $CSMDATA ccsm2.0.tar.gz (35 Mbytes) Scripts & Model Code ccsm2.0.inputdata.T42_gx1v3.tar (.8 Gbytes ) Initial data

6. www.ccsm.ucar.edu The CCSM source code: inputdata/

7. www.ccsm.ucar.edu The CCSM source code: Top Level ($CSMROOT)

8. www.ccsm.ucar.edu The CCSM source code: models/

9. www.ccsm.ucar.edu The CCSM source code: scripts/ test.a1.run cpl.setup.csh atm.setup.csh ocn.setup.csh ice.setup.csh lnd.setup.csh datm.setup.csh docn.setup.csh dice.setup.csh dlnd.setup.csh ccsm_archive test.a1.har ccsm_checkenvs ccsm_cpdata ccsm_getfile ccsm_getinput ccsm_getrestart ccsm_msmkdir ccsm_msread ccsm_mswrite ccsm_splitdf modules.AIX.seaborg modules.IRIX64.nirvana modules.IRIX64.ute test.a1.mods.* ccsm_gui.pl

10. www.ccsm.ucar.edu Building the CCSM Three levels of c-shell scripts are used: –Run script builds & runs CCSM ( i.e. test.a1.run ) –Define the CCSM environment –Run each component setup script –Execute all components simultaneously –Setup scripts builds each model ( i.e. ice.setup.csh ) –Position any input data needed by that component –Generate the component's namelist input file –Build the component executable –Utility tools ( i.e. ccsm_getinput )

11. www.ccsm.ucar.edu Building the CCSM A sample run script (test.a1.run) is distributed. Functional summary of test.a1.run setenv A abc setenv B xyz foreach i (cpl atm ocn ice lnd) $i.setup.csh end mpirun cpl atm ocn ice lnd exit Loop over setup scripts Define environment Run CCSM

12. www.ccsm.ucar.edu Building the CCSM: $CASE $CASE identifies a CCSM run. The sample $CASE is test.a1, $CASE for the CCSM control run is b20.007 To define a new $CASE –Copy all of test.a1/ into $CASE/ –rename test.a1.run to $CASE.run –Modify the main run script –CASE CASESTR CSMROOT CSMDATA EXEROOT ARCROOT –$HOME/ccsm2.0/scripts/$CASE/$CASE.run submitted to the batch queue.

13. www.ccsm.ucar.edu Building the CCSM $HOME/ccsm2.0/ $CSMROOT /fs/$LOGANME/inputdata/ $CSMDATA Scripts & Model Code Initial data /ptmp/$LOGNAME/$CASE $EXEROOT Large model build/execute directories doc/models/scripts/

14. www.ccsm.ucar.edu Methods for starting the CCSM New CCSM case is started as either a startup, hybrid or branch run, depending on the science requirements. startup:new run from arbitrary input files hybrid:new run from initial or restart files branch:new run from restart files continue: extend an existing run exactly

15. www.ccsm.ucar.edu General concepts: Exact Restarts Wall-clock limits in the batch queues typically restrict runs to a finite length, usually 1-3 model years. At specified intervals, restart and initial files will be created by all components Runs are extended as "continue" runs. A continued run must give exactly the same results as if the run had never stopped.

16. www.ccsm.ucar.edu Are your results reproducible? Factors: Any change in machine architecture Any change in math libraries Running on different number of processors Changes you've made to the code Additional variables added to model not properly written to restart files Exact Restarts

17. www.ccsm.ucar.edu Changing the code Frozen Code Keep the CCSM2.0 model code "frozen" To introduce code changes: copy specific model files to new directory modify the new copies of model code tell the model setup scripts to get the modified code from the new directory first, then get rest of unchanged model code from the "frozen" version This method encapsulates modified code Use same "frozen code" for different work

18. www.ccsm.ucar.edu Changing the code Three levels of change: 1. "Bit-for-bit" (no change to answers) adding/subtracting output 2. Machine-level (answers similar at the bit level) Changes in operation order Changes in math libraries changes in compiler options/flags/versions 3. Algorithm changes (answers change) Different parameterization Rerun control integration

19. www.ccsm.ucar.edu atm ocn ice lnd Coupler CCSM Components

20. www.ccsm.ucar.edu CCSM2: Performance OCN ICE LND ATM CPL 8 3 7 7 4 3 7 2 14 41secs 53 secs/day total (~130 years/month) NERSC T42/gx1v3 (128Pes) 32 8 8 48 Critical path

21. www.ccsm.ucar.edu Output log files The printed output from each CCSM component is saved in a "log file" A coordinated timestamp links filenames $LID is of the form YYMMDD-hhmmss, where YYMMDD are the Year, Month, Day and hhmmss are hour, minute and second that the run began (i.e. $EXEROOT/ocn/ocn.log.000626- 082714).

22. www.ccsm.ucar.edu History and restart output data Default: netCDF monthly averaged history files for all components in netCDF format. CCSM2.0 also writes out binary restart files from all components at regular intervals. The total output volume of the model output can vary greatly depending upon the output frequencies and options selected.

23. www.ccsm.ucar.edu Tape Archive Device cpl atm ocn ice lnd 1.4 Gbytes 0.7 Gbytes 4.1 Gbytes.6 Gbytes.4 Gbytes disk Tape Archive Devices 7.4 Gbytes/year CCSM Yearly Output (b20.014)

24. www.ccsm.ucar.edu Getting help http://www.ccsm.ucar.edu forum ccsm@ucar.edu