1. WRF4G The Weather Research Forecasting model workflow for the GRID meteo@unican.es Department of Applied Mathematics & Computer Sciences University of Cantabria, Spain GRID developers: Valvanuz Fernández, Antonio S. Cofiño Application developers: Jesús Fernández, Lluís Fita Santander Meteorology Group A multidisciplinary approach to weather & climate http://www.meteo.unican.es EGU General Assembly 2010, Wien, 2-7 May 2010

2. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es Numerical Weather Prediction model (regional)‏ Open source (Fortran90)‏. A world community with 6000+ registered users. Application in a wide variety of studies:  Weather forecasting ( operational at NCEP )‏  Data assimilation studies  Regional climate studies  Idealized simulations  Educational applications,.... WRF Model

3. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es Computing and storage requirements:  Intensive use of CPU  Supports a variety of parallel programming paradigms (OpenMP, MPI, serial,...). It's a common benchmarking application for HPC.  Large amount of input, output and restart data.  Typical experiments last for days (even in parallel).  Application with a complex workflow (preprocessing, execution & postprocessing)‏ WRF Requirements metgridwrf geogrid ungrib WRF Preprocessing System WRF ARW WRFGEL namelist.input namelist.wps real WRF WORKFLOW

4. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es Reanalisys/Reforecasting  High number (~10 4 ) of independent simulations  High volume of output-data (>TB)  Requires scalability Regional projections for climate change  Contiguous simulations (~10), weeks of walltime each.  High volume of output data (>10TB)  Recovering system for simulation restart Weather Forecasting  QoS and optimal resources: deadline for delivering Sensitive studies for Climate and Weather  Physical schemes, initial conditions and boundary conditions uncertainties sampling  Resource demanding experiments Scientific experiments WRF

5. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es GOALS  Make easier for the user the process of design, execution and monitoring of the experiments with WRF.  Develop a framework that allow the user to use at the same time different computing resources in a transparent way: Local clusters (PBS, SGE, LoadLeveler,…) Grid infrastructures (gLite, Globus,…) Local resources (SSH, fork, …).  Develop a set of command line tools and a Web portal for the WRF users. WRF4G: Goals

6. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es CORDEX (COordinated Regional climate Downscaling EXperiment) is a framework to improve coordination of international efforts in regional climate downscaling research. CORDEX was initiated as a result of the Task Force on Regional Climate Downscaling, formed by the World Climate Research Program (WCRP). A set of target regions has been proposed and modeling groups willing to contribute must comply with simulations specifications. CORDEX will produce an ensemble of simulations sampling uncertainties related to: (i) varying Global Climate Model (GCM) simulations; (ii) varying greenhouse gas (GHG) concentration scenarios; (iii) natural climate variability; and (iv) different downscaling methods. http://wcrp.ipsl.jussieu.fr/RCD_Projects/CORDEX http://www.meteo.unican.es/wiki/cordexwrf There are currently 15 groups planning to contribute to CORDEX with WRF, 5 groups planning to contribute the African domain (key region for the AR5). All of them could benefit from WRF4G app. Role of WRF4G in CORDEX

7. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es 150.000 CPUs 70 PB 260 sites worldwide Arquitectures: i386,x86_64 LRMS: torque, sge, lsf, bqs Shared and not shared Home Computing resources:  OS: Debian, SL/Centos 4 and 5.  Memory: 250MB to 16GB  Processor: P4 to i7 Different queues limitations: walltime, memory & disk quotas… Bandwidth in some sites is very small. EGEE: Example of big Grid infrastructure

8. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es GRID Challenges:  Develop an application adapted to run in different arquitectures, OS, LRMS and parallel environments.  Repository of WRF binaries adapted to those environments  Detect sites that not accomplish application requirements: Memory, Processor, queue time limitations,…  Optimizing the data transfers and replication between resources distributed geographically.  Develop a monitoring tool for experiments status.  Create a checkpointing management system that allow to restart simulations.  Failure detection and recovery. WRF4G: GRID Challenges

9. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es WRF4G is a port of the WRF Modeling System prepared to run in GRID environments. WRF has been splitted in 2 layers to separate the scientific experiment from the infrastructure details (wrf.input & wrf4g.conf). WRF4G supports several data transfer services (gsiftp, rsync,…) and execution systems (gLite, globus, PBS, SGE,…). It can be run in the user’s machine. No deployment required on sites. The application is deployed by a pilot job. WRF for GRID: WRF4G

10. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es metgridrealwrf geogrid Static Geographical Data ungrib Gridded Data: NAM, GFS, RUC, AGRMET, etc. WRF Preprocessing SystemWRF ARW namelist.inputnamelist.wps WRF4G Workflow

11. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es metgridrealwrf geogrid Static Geographical Data ungrib Gridded Data: NAM, GFS, RUC, AGRMET, etc. WRF Preprocessing SystemWRF ARW Processed locally wrf4g.conf wrf.input namelist.inputnamelist.wps WRF4G WRF4G Workflow

12. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es metgridrealwrf geogrid Static Geographical Data ungrib Gridded Data: NAM, GFS, RUC, AGRMET, etc. WRF Preprocessing SystemWRF ARW WRFGEL WRF4G.sh Processed locally WRF4G_ini.sh wrf4g.conf wrf.input namelist.inputnamelist.wps wrf4g_submitter.sh Job submission abstraction layer WRF4G UI WN Input data abstraction layer WRF4G Workflow

13. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es metgridrealwrf geogrid Static Geographical Data ungrib Gridded Data: NAM, GFS, RUC, AGRMET, etc. WRF Preprocessing SystemWRF ARW WRFGEL WRF4G.sh Processed locally WRF4G_ini.sh wrf4g.conf wrf.input namelist.inputnamelist.wps wrf4g_submitter.sh Job submission abstraction layer WRF4G UI WN WRF4G-0.0.2.tgz WRF4Gbin-3.1r83.tgz wrfout_* wrfrst_* wrfinput_* wrfbdy_* WRF4G_BASEPATH WRF4G_INPUT WRF4G_APPS Input data abstraction layer geo_em.nc namelist.wps WRF4G_DOMAINS WRF4G Workflow

14. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es ${WRF4G_ROOT} scripts uiwn WRFGEL WPSWRFV3 binlib examples WRF4G_ini.sh wrf4g_submitter.sh wrf4g_make_tarball.sh wrf4g_make_tarball_bin.sh wrf4g_submit.EELA_grid_job wrf4g_submit.MDMcluster wrf4g_submit.MDMclusterIFB wrf4g_wrfgel_environment create_output_structure download_file exist_wps get_date_restart register_file cccantabria wrf.input wrf4g.conf domain_partition wrf.input.in wrf4g.conf.in nino50 wrf.input wrf4g.conf operativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf vcp fortnml ncdump preprocessor.* WRF4G.sh WRF4G structure

15. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es These are sample configuration files for a WRF4G experiment ${WRF4G_ROOT} scripts uiwn WRFGEL WPSWRFV3 binlib examples WRF4G_ini.sh wrf4g_submitter.sh wrf4g_make_tarball.sh wrf4g_make_tarball_bin.sh wrf4g_submit.EELA_grid_job wrf4g_submit.MDMcluster wrf4g_submit.MDMclusterIFB wrf4g_wrfgel_environment create_output_structure download_file exist_wps get_date_restart register_file cccantabria wrf.input wrf4g.conf domain_partition wrf.input.in wrf4g.conf.in nino50 wrf.input wrf4g.conf operativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf vcp fortnml ncdump preprocessor.* WRF4G.sh WRF4G structure

16. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es These are user scripts launched from the UI ${WRF4G_ROOT} scripts uiwn WRFGEL WPSWRFV3 binlib examples WRF4G_ini.sh wrf4g_submitter.sh wrf4g_make_tarball.sh wrf4g_make_tarball_bin.sh wrf4g_submit.EELA_grid_job wrf4g_submit.MDMcluster wrf4g_submit.MDMclusterIFB wrf4g_wrfgel_environment create_output_structure download_file exist_wps get_date_restart register_file cccantabria wrf.input wrf4g.conf domain_partition wrf.input.in wrf4g.conf.in nino50 wrf.input wrf4g.conf operativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf vcp fortnml ncdump preprocessor.* WRF4G.sh WRF4G structure

17. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es This is what the Worker Node sees just before running WRF4G.sh ${WRF4G_ROOT} scripts uiwn WRFGEL WPSWRFV3 binlib examples WRF4G_ini.sh wrf4g_submitter.sh wrf4g_make_tarball.sh wrf4g_make_tarball_bin.sh wrf4g_submit.EELA_grid_job wrf4g_submit.MDMcluster wrf4g_submit.MDMclusterIFB wrf4g_wrfgel_environment create_output_structure download_file exist_wps get_date_restart register_file cccantabria wrf.input wrf4g.conf domain_partition wrf.input.in wrf4g.conf.in nino50 wrf.input wrf4g.conf operativo operativo.sh post_operatorio.sh wrf.input.in wrf4g.conf vcp fortnml ncdump preprocessor.* WRF4G.sh WRF4G structure

18. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es User Interface - my_experiments/exp wrf4g.conf wrf.input Create and run an experiment

19. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es User Interface - my_experiments/exp exp__...exp__reaN 00010002...00010002...00010002... wrf4g.conf wrf.input sandbox.tgz wrf.chunk wrf.input wrf4g.conf WRF4G_ini exp__rea1 realizations Create and run an experiment

20. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time Three realizations split into two chunks each: exp__rea1 exp__rea2 exp__rea3 Storage Element - ${WRF4G_BASEPATH}/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 Create and run an experiment

21. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 wrfinput_t0 wrfbdy_t0 Three realizations split into two chunks each: Create and run an experiment

22. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 wrfout_t0 wrfout_t1 wrfinput_t0 wrfbdy_t0 Three realizations split into two chunks each: Create and run an experiment

23. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 wrfout_t0 wrfout_t1 wrfout_t2 wrfrst_t2 wrfinput_t0 wrfbdy_t0 Three realizations split into two chunks each: Create and run an experiment

24. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfrst_t2 wrfrst_t4 wrfinput_t0 wrfbdy_t0 Three realizations split into two chunks each: Create and run an experiment

25. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfrst_t2 wrfrst_t4 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

26. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfrst_t2 wrfrst_t4 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

27. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfrst_t2 wrfrst_t4 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

28. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout 1230456 wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfout_t6 wrfout_t7 wrfout_t8 wrfrst_t2 wrfrst_t4 wrfrst_t6 wrfrst_t8 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

29. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es time exp__rea1 exp__rea2 exp__rea3 Storage Element - METEO4G/WRF/experiments/exp exp__rea1exp__rea2exp__rea3 outputrestartwpsoutoutputrestartwpsoutoutputrestartwpsout wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfout_t6 wrfout_t7 wrfout_t8 wrfrst_t2 wrfrst_t4 wrfrst_t6 wrfrst_t8 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 1230456 wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfout_t6 wrfout_t7 wrfout_t8 wrfrst_t2 wrfrst_t4 wrfrst_t6 wrfrst_t8 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 wrfout_t0 wrfout_t1 wrfout_t2 wrfout_t3 wrfout_t4 wrfout_t5 wrfout_t6 wrfout_t7 wrfout_t8 wrfrst_t2 wrfrst_t4 wrfrst_t6 wrfrst_t8 wrfinput_t0 wrfbdy_t0 wrfinput_t4 wrfbdy_t4 Three realizations split into two chunks each: Create and run an experiment

30. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es To hide the complexity of the GRID to the application developer, WRF4G includes several abstraction layers which encapsulate the routine tasks in generic function calls:  Data Management vcp: Provides transparent copy between any of the following protocols: gsiftp, LFC, rsync, local copies vcp -r /local/dir gridftp://server:port/remote/path vcp gsiftp://srv1:port/rmt/file gsiftp://srv2:port/other/file vcp /local/file1 rsync://server/other/local/path/ vcp gsiftp://srv1:port/rmt/file ln:link WRF4G architecture (I)‏

31. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es To hide the complexity of the GRID to the application developer, WRF4G includes several abstraction layers which encapsulate the routine tasks in generic function calls:  Execution Management Through the use of plugins, the user can submit jobs to a different infrastructures: Grid (globus 4.2 and glite CE), local clusters (PBS, SGE, SLURM,…), local resources. Prepared to use different execution environments: OpenMP and MPI. Strong scheduling policy based in history records and resources characteristics. Failure detection and recovery. WRF4G architecture (II)‏

32. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es The application is been supported in EELA-2, EGEE & NGI-ES infrastructures and GT2.4 & GT4. A release candidate (v1.0 RC1) has been launched to run realistic experiments. It consist on a virtual machine UI with all the application components. Collaborations with end-users in LA ( CETA-CIIFEN- UPS ), Europe ( CESGA-MeteoGalicia ) for feedback and Asia ( HAII, Thailand ). Currently we are testing MPI support in the GRID. Daily operational version running on the GRID for ensemble of weather forecasts. Conclusions & Current Work

33. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es Develop a monitoring system oriented to app status. Execution management works on Globus and gLite CE based Gridway plugins. PBS y SGE are used, but plugins will developed for better integration with Gridway. Develop an scalable replica management integrated with Gridway. (bottleneck in GRID infrastructures). Create a user portal that allow the users managing and monitoring their experiments. Collaborate with EGI and other International GRID Initiatives like EUAsiaGrid Incorporate users feedback Future Work

34. A multidisciplinary approach to weather & climate Santander Meteorology Group http://www.meteo.unican.es Thanks !!! meteo@unican.es