1. 1 CCOS Seasonal Modeling: The Computing Environment S.Tonse, N.J.Brown & R. Harley Lawrence Berkeley National Laboratory University Of California at Berkeley

2. 2 Overview Modeling Approach Linux Cluster: OS & hardware Parallel computing & performance CMAQ flavours and features

3. 3 Modeling Approach MM5 model provides 3D gridded spatial/temporal wind vectors, temperature fields, cloud cover etc. MM5 code run by ETL group of NOAA (J.Wilczak et al) CMAQ (Community Multiscale Air Quality) Model (US EPA) incorporates meteorology results from MM5 with emissions. Fortran 90. Runs on Linux platforms with Pentiums or Athlons. 3D time-stepping grid simulation includes atmospheric chemistry and advective and diffusive transport. Provides concentration for each 4km x 4km grid cell at every hour.

4. 4 Modeling Approach Meteorological Model (MM5) simulation of week-long ozone episode Other emissions: motor vehicle, biogenic, major point sources, and area Photochemical AQM: (CMAQ). Ozone Transport, Chemistry, Sources and Sinks Scenarios of Central Utility Generator emissions

5. Sarmap Grid inside CCOS Grid CCOS: 190 x 190 SARMAP 96 x 117

6. The Mariah Linux Cluster: Hardware and OS Purchased with DOE funds Maintained by LBNL under the Scientific Cluster Support Program. 24 nodes, each has 2 Athlon processors, 2GB RAM, see eetd.lbl.gov/AQ/stonse/mariah/ for more pictures. Centos Linux (similar to Red Hat Linux), run in a Beowulf- cluster configuration

7. 7 Parallel Computing on Cluster (1) Typical input file sizes for a 5-day run: Meteorological files: 2.7 GigaBytes, Emission file: 3 Gigabytes. Typical output file sizes : Hourly output for all (or selected) grid cells, for (1) Concentration file of selected species: ~2GB (2) Process analysis files, i.e. analysis of the individual contributions from CMAQ’s several physical processes and the SAPRC99 mechanism’s numerous chemical reactions: (2-4 GB)

8. 8 Parallel Computing on Cluster (2) Typically split the 96x117 grid 3 ways in each direction at beginning of run and use 9 processor elements (PE’s) Message Passing Interface (MPI) sends data between PE’s to handle the needs of a PE for data from a neighbouring sub-domain. MPI subroutine calls embedded in CMAQ code A 5-day run takes about 5 days with a stiff Gear solver Takes about 1 day with backward Euler solver (MEBI) hard-coded for calculations of the SAPRC99 mechanism Often we have to use the Gear solver. Can we use more PE’s to speed up?

9. 9 Parallel Computing on Cluster (3) Performance of CMAQ as number of Processor increases: No. PE’sTime (s)ScalabilityEffective no. PE’s 14500100%1 2243192%1.84 6101974%4.44 1273950%6 2453935%8.4

10. 10 Parallel Computing on Cluster (4) Computational load is not balanced between PE’s, as geographical locations with more expensive chemistry use more CPU time. Code only runs as fast as PE with greatest load. Others must wait for it at “barriers”. As no. of PE’s increases, probability of larger discrepancies increases.

11. 11 Parallel Computing on Cluster (5) More PE’s  sub-domains decrease in size,  relatively more expense toward inter-processor communication Key parameter is ratio between a sub-domain’s Perimeter/Area. Cost of scientific calculation depends on Area increases which as N 2 Cost of communication depends on Perimeter which increases as N

12. 12 CMAQ Flavours (1) Sensitivity Analysis, via Decoupled Direct Method CMAQ DDM4.3, for sensitivity of any output to: Emissions (from all or part of the grid) Boundary Conditions Initial Conditions Reaction rates (not implemented yet) Temperature (not implemented yet) Process Analysis: Analyse internal goings-on during course of a run, eg. Change in O 3 in a cell due to vertical diffusion or to a particular chemical reaction. All versions of CMAQ cannot do all of these things

13. CMAQ Flavours (2) CMAQ Version SolverSensitivityProcess Analysis Comments 4.3GEARNoYesSlow 4.3MEBINo Fast. Inaccurate for minor species DDM 4.3MEBIYesNo Fast. Inaccurate for minor species 4.4GEARNoYesSlow 4.4EBINo Fast X.x???Yes Fast

14. Last Slide Here.