2. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke1 Session 1: Agenda Integrating Model and Data Infrastructure Overview  History  High level design Source Code Management (SCM) Standard Compile Environment (SCE) Standard Running Environment (SRE)

3. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke2 IMDI: History Develop for the European Earth System (ES) modelling community a software infrastructure to compile run set up analyse coupled ES model experiments and thereby establish a European ES research network Since 2005: cooperation with PRISM Support Initiative (PSI) see http://prism.enes.orghttp://prism.enes.org continued development of SCE/SRE integration of data aspects (WDCC, data proc./vis.) EU FP5 Project for Integrated Earth System Modeling (PRISM) Dec 2001-Nov 2004:

4. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke3 IMDI: High Level Design Create a common infrastructure for ES modeling and data Modularity:  all components of IMDI independent Interoperability:  all components of IMDI work together Modularity in terms of experiment tasks Common look&feel for all (coupled) models and platforms Integrate best practices Modularity in terms of ES components

5. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke4 IMDI: Benefits Facilitates exchange of component models  between modellers  between institutes communication between model users exchange of results intercomparison of simulations Common look&feel: learn only once Choose what you like/need

6. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke5 SCM/SCE: Work Flow Source & Script Repository (SVN) Update Makefiles Modify source code Retrieve source & script code Adapt source to SCE Create Compile Scripts Compile model.x

7. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke6 Use tools for SC analysis with[out] SCE  list header file suffixes  list directories containing header files  list VPATH directories  detect multiple file names  multiple MODULEs per file detection  MOULE name != file name detection Use prerequisite files with[out] SCE Makefiles Use Makefile with[out] SCE compile script SCM/SCE: Modular but Interoperable

8. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke7 SCE/SRE: Independent but Interoperable Adapt I/O to SRE Retrieve SRE code Script code Repository (SVN) Create Exp. Scripts Configure setup file Launch Exp. model.x SCE

9. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke8 Postprocessin g DB-Filling LE-Visualisation HE-Visualisation File Archive Preprocessing Start Rerun Stop DB Archive ww w Archiving Postprocessin g DB-Filling LE-Visualisation HE-Visualisation File Archive Preprocessing Start Rerun Stop DB Archive ww w Archiving Post- processing DB-Filling LE- Visualisation Pre processing Start Rerun Stop ww w File Archive HE- Visualisation Processing SRE-tasks: Modularity by Tasks Archiving

10. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke9 Create Compile scripts Compile Create Run Scripts Run Experiment user SCE/SRE model_tmpl.xsd model.xml tmpl.xsd model_tmpl.xsd model.xml SCE/SRE/DataManagement: Modular but Interoperable model.xml

11. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke10 SCE / SRE: Modularity by ES Components Applications: PRISM (coupled) models (~12 components in 10 combinations on 10 platforms) COSMOS model family of MPI-Met (cosmos-a, cosmos-as, cosmos-ao, cosmos-aob, cosmos- asob, cosmos-aso, cosmos-ob, cosmos-o)

12. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke11 ECHAM5 JSBACH MPIOM Surface fluxes Surface conditions CO2 Grid transformation Synchronisation OASIS3 HAMOCC CO2 cosmos-a cosmos-as cosmos-o cosmos-ob cosmos-asob

13. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke12 cosmos-ao ECHAM5 MPIOM Surface fluxes Surface conditions Grid transformation Synchronisation OASIS3 HAMOCC CO2 JSBACH CO2 cosmos-aobcosmos-asob

14. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke13 Session 1: Agenda Integrating Model and Data Infrastructure Overview Source Code Management (SCM)  SVN repository  Source/script code directory structure  Browsing facility (being revised)  Scripts to analyse and transform source code Standard Compile Environment (SCE) Standard Running Environment (SRE)

15. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke14 SCM: Modularity by ES components & libraries Entities: the OASIS3/4 coupler interpolation and synchronisation component models source code packages describing sub systems of the ES (e.g. sea ice, ocean, clouds,...) main models: components which create an executable (e.g. MPIOM) submodels: component (optionally) linked to a main model (e.g. HAMOCC in MPIOM) libraries source code package not representing subsystems of the ES

16. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke15 SCE: Libraries and Models Components: Configurable => Can not be precompiled OASIS3/4 Not configurable => Can be precompiled Not representing a subsystem of ES Libraries: Not configurable => Can be precompiled Potentially usable by all components

17. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke16 USE, CALL, INCLUDE SVN/SCM: Source Code Directory src ok ok for call of SBR if mod2 is submodel of mod1

18. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke17 SCM: Restructure_directories.ksh tools/Restructure_directories.ksh

19. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke18 Source code conventions: no two files with same base name (file.f90 / file.c) one MODULE per file only MODULE name = file name Use Coding_conventions.ksh for renaming / splitting in last two cases  enables use of Create_prerequisites... No change of source code proper required ! SCM: Coding_conventions.ksh

20. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke19 SCM/SCE/SRE: Implementation Stand-alone models Unix-ksh (g)make m4, perl Coupled models OASIS 3/4 (http://prism.enes.org) OASIS model interface library PSMILe; NetCDF library; message passing library MPI-1/2; I/O library mpp_io => use help function: script.ksh --help => get help from model@dkrz.de

21. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke20 Session 1: Agenda Integrating Model and Data Infrastructure Overview Source Code Management (SCM) Standard Compile Environment (SCE)  SCE toolkit: Support for Makefile writing (templates) Creating compile scripts Maintaining Makefiles (prerequisites) Standard Running Environment (SRE)

22. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke21 SCE/SRE: Directories

23. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke22 SCE/SRE: Toolkit Directory util

24. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke23 SCE: Toolkit Directory util/compile

25. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke24 No compile scripts are provided … … but tools that create them … … for the specific platform … for the specific component model or library … configured for the coupled combination of models  common look&feel SCE: Compile Toolkit

26. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke25 Create_COMP_cpl_models.ksh: create all compile scripts for a coupled model (coupler, components, libraries) Create_COMP_models.frm: create compile scripts for a component model or the coupler Create_COMP_libs.frm: create compile scripts for the libraries Create_prerequisites create a file containing USE or include based prerequisites Create_prerequisites_cpp create a prerequisite file controlling recompilation if cpp flags have changed SCE: Compile Toolkit

27. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke26 SCE: Creating compile scripts Create_COMP_cpl_models.ksh cplmod \ [--id=ID [-n=node [-s=MPI1 [-c=sun [--stdout=“-”]]]] Create_COMP_models.frm mod1 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3” Create_COMP_models.frm mod2 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3” Create_COMP_models.frm mod3 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3” Create_COMP_models.frm oasis3 -s=MPI1 -nnode Create_COMP_libs.frm -s=MPI1 -nnode COMP_mod1_ID.ksh COMP_mod2_ID.ksh COMP_mod3_ID.ksh COMP_oasis3_MPI1.ksh COMP_libs.ksh cplmod = mod1/mod2 & mod3 & oasis3: mod1_mod2_ID.MPI[1,2].x mod2_ID.MPI[1,2].a mod3_ID.MPI[1,2].x oasis3.MPI[1,2].x …., psmile.MPI[1,2].a mod1_mod2_ID.MPI[1,2].x mod2_ID.MPI[1,2].a mod3_ID.MPI[1,2].x oasis3.MPI[1,2].x …., psmile.MPI[1,2].a mod1_mod2_ID.MPI[1,2].x mod2_ID.MPI[1,2].a mod3_ID.MPI[1,2].x oasis3_MPI[1,2].x …., psmile.MPI[1,2].a

28. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke27 SCE: Creating Compile Scripts util/ compile/ headers/ incl*/*.h Compile scripts for - libraries (1 for all) - component mod - coupler m4... include incl_mod_$model/GUIspecif_$model.h include include_$node/Sitespecific_$node.h include incl_mod_$model/Cppflags_$model.h include include_$node_$model/Compile_mode_$model_$node.h include Make_model.h … Create_COMP_models.frm model –nnode

29. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke28 cosmos-aob = echam5 & mpiom+hamocc & oasis3 Create_COMP_cpl_models.ksh cosmos-aob COMP_echam5_ID.ksh COMP_mpiom_ID.ksh default ID COMP_hamocc_ID.ksh COMP_oasis3_MPI1.ksh COMP_libs.ksh cosmos-a = echam5 cosmos-as = echam5+jsbach cosmos-ao = echam5 & mpiom & oasis3 cosmos-aob = echam5 & mpiom+hamocc & oasis3 cosmos-asob = echam5+jsbach & mpiom+hamocc & oasis3 cosmos-aso = echam5+jsbach & mpiom & oasis3 cosmos-ob = mpiom+hamocc cosmos-o = mpiom SCE: Creating compile scripts (examples) [--id=ds]

30. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke29 Creating compile scripts: examples clm = clm Create_COMP_cpl_models.ksh clm COMP_clm_ID.ksh toyclim = toyatm & toyoce & toyche & oasis3 Create_COMP_cpl_models.ksh toyclim COMP_toyatm_ID.ksh COMP_toyoce_ID.ksh COMP_toyche_ID.ksh COMP_oasis3.ksh COMP_libs.ksh

31. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke30 Libraries lib*.a Libraries lib*.a Compile libs of main start Compile sub model Compilation: Main+Sub Model Compile libs of sub executable Compile main model

32. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke31 mpiom_hamocc_ID[.MPIn].x n = 1 or 2 COMP_hamocc_ID.ksh COMP_libs.ksh support hamocc / libhamocc_ID[.MPIn].a mpiom / libmpiom_hamocc _ID[.MPIn].a Local libsupport.a Central start COMP_mpiom_ID.ksh Compilation: Main+Sub Model with central libraries COMP_libs.ksh support

33. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke32 *.o *.mod *.a modn *_ID.a *_ID.x *.o *.mod SCE: Architecture Dependent Directories

34. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke33 Model compilation *.o / *.mod mod1_ID.x lib*.a make src directories: make dir/Makefile Top_Makefile set / export par. create (in-doc.) COMP_mod1_ID.ksh all / lib / clean / tar library list: make src / Makefile *.o / *.mod Top_Makefile make set / export par. create (in-doc.) COMP_libs.ksh “library list” with ‚all‘ only

35. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke34 All compilation is based on the (g)make software: Make ‘targets’ having ‘prerequisites’ with well defined ‘rules’ while avoiding redundant actions Targets, rules & prerequisites are defined in file Makefile: SCE: (GNU) Make … target: prerequisite1, prerequisite2 …. rules … ‘what to make’: ‘what has to be made before’ ‘how to make’ …

36. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke35 SCE Portable Makefiles make [–f Makefile] All non portable code is exported from the compile scripts 1 Makefile for each source directory with compiler input (*.F90, *.f90, *.F, *.f, *.c) All Makefiles have the same look and feel Prerequisites Tool for generation of *.o, *.h prerequisites Tool for generation of cpp flag prerequisites

37. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke36 SCE Make Targets Executable: OASIS3_MPI1/2.x, ECHAM5_ID[_MPI1/2].x, *.x Libraries: libblas.a, libsupport.a, libpsmile.MPI1/2.a, lib*.a Model libraries: hamocc_ID[_MPI1/2].a, *_ID.a Binary object code: *.o Empty build directory tar file of model code

38. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke37 SCE make prerequisites Executable: Libraries and model libraries echam5_ID.x: libsupport.a mpiom_ID.x: libmpiom_ID.a Library: object binaries libsupport.a: *.o Binary objects: F90 MODULEs and include files file.o: module.mod (aliased to module.o) file.o: include.h file.o: file.F90  Note: with multiple target lines for the same target: the join of all prerequisite sets is valid

39. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke38 SCE make rules $(PROG): $(LIBRARY) $(F90) master.o echam5_ID.a support.a –o echam5_ID.x $(LIBRARY): *.o $(AR) libsupport.a *.o %.o: %.F90 $(F90) -c $< file.o: file.F90 $(F90) -c $<  Note: with multiple target lines for the same target: the last rule is valid

40. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke39 Example Makefile: Targets & Rules clean: rm -f $(LIBRARY) ls | xargs rm – f lib: $(LIBRARY) all: $(PROG) $(PROG): $(DEPLIBS) $(LIBRARY) $(F90) $(LDFLAGS) -o $@ $(MAINPRG).o $(LIBS) $(LIBRARY): $(OBJS1) $(OBJS2) $(OBJS3) $(OBJS4) $(OBJS5) $(AR) $(ARFLAGS) $(LIBRARY) *.o.SUFFIXES:.SUFFIXES:.o.c.f.F.f90.F90 %.o: %.F90 $(F90) $(F90FLAGS) $(INCLS) -c $<... %.o: %.c $(CC) $(CCFLAGS) $(INCLSC) -c $< include../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.deps -include../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.cpp -include../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.special -include../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.$(ARCH) Create_prerequisites[_cpp]

41. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke40 Example Makefile: Definition of variables SRCS1 = $(shell \ find../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)) – name '*.F90 ‘ print) OBJS1 = $(patsubst../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/%.F90, %.o, $(SRCS1))... SRCS5 = $(shell \ find../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)) -name '*.c' -print) OBJS5 = $(patsubst \../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/%.c, %.o, $(SRCS5)) VPATH =.:\../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)):\../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/include:\../../../../src/mod/$(MODEL_DIR)/include:\../../lib/psmile.${CHAN}:\ $(NETCDF_INCLUDE):$(MPI_INCLUDE):$(SYS_INCLUDE):\../../../lib/$(MODEL_DIR):\../../../lib/$(SUBMOD1):\../../../lib:\../../../../src/mod/include/make_dummies LIBRARY =../../../lib/$(MODEL_DIR)/lib$(MODLIB).a PROG =../../../bin/$(EXEC).x INCLS = -I../../../../src/mod/$(MODEL_DIR)/include \ -${I4mods}../../lib/clim.${CHAN} INCLSC = -I../../../../src/mod/$(MODEL_DIR)/include

42. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke41 Make.inc.deps mod_unitncdf.o: mod_kinds_oasis.o alloc_src.o: mod_anais.o mod_analysis.o mod_coast.o mod_experiment.o \ mod_extrapol.o mod_kinds_oasis.o mod_memory.o mod_nproc.o \ mod_parallel.o mod_parameter.o mod_pipe.o mod_rainbow.o mod_sipc.o \ mod_string.o mod_timestep.o mod_unitncdf.o dealloc_src.o: mod_anais.o mod_analysis.o mod_coast.o mod_experiment.o \ mod_extrapol.o mod_kinds_oasis.o mod_memory.o mod_nproc.o \ mod_parallel.o mod_parameter.o mod_pipe.o mod_rainbow.o mod_sipc.o \ mod_string.o mod_timestep.o mod_unitncdf.o extrap.o: mod_extrapol.o mod_kinds_oasis.o mod_parameter.o mod_printing.o \ mod_unit.o getfld.o: mod_analysis.o mod_clim.o mod_experiment.o mod_hardware.o \ mod_kinds_oasis.o mod_label.o mod_memory.o mod_parameter.o \ mod_printing.o mod_sipc.o mod_string.o mod_timestep.o mod_unit.o \ mod_unitncdf.o netcdf.inc Create_prerequisites [model,library] directory

43. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke42... mod_unitncdf.o: use_netCDF.cpp alloc_src.o: use_comm_MPI1.cpp use_comm_MPI2.cpp use_comm_GSIP.cpp dealloc_src.o: use_realtype_single.cpp... Make.inc.cpp Create_prerequisites_cpp [model,library]

44. 29 March 2007DKRZ / M&D Benutzerschulung, Stephanie Legutke43 SCE Handbook: SVN: http://svn-mad.zmaw.de/svn/mad/ \ Model/IMDI/tags/’tag’/util/compile/doc http://svn-mad.zmaw.de/svn/mad/ \ Model/IMDI/tags/’tag’/util/compile/doc Web: http://mad.zmaw.de/imdi (bottom of page): http://mad.zmaw.de/imdi  - pdf-version  - html-version Handbooks