1. August 2001 Parallelizing ROMS for Distributed Memory Machines using the Scalable Modeling System (SMS) Dan Schaffer NOAA Forecast Systems Laboratory (FSL) August 2001

2. August 2001 Outline Who we are Intro to SMS Application of SMS to ROMS Ongoing Work Conclusion

3. August 2001 Who we are Mark Govett Leslie Hart Tom Henderson Jacques Middlecoff Dan Schaffer Developing SMS for 20+ man years

4. August 2001 Intro to SMS Overview –Directive based FORTRAN comments Enables single source parallelization –Distributed or shared memory machines –Performance portability

5. August 2001 Distributed Memory Parallelism

6. August 2001 Add SMS Directives Code Parallelization using SMS SMS Serial Code SMS Parallel Code Original SerialCode PPP Parallel Pre-Processor Serial Executable Parallel Executable

7. August 2001 Low-Level SMS MPI, SHMEM, etc. NNTSRSSST FDA Library Spectral Library Parallel I/O SMS Parallel Code

8. August 2001 Intro to SMS (contd) –Support for all of F77 plus much of F90 including: Dynamic memory allocation Modules (partially supported) User-defined types –Supported Machines COMPAQ Alpha-Linux Cluster (FSL “Jet”) PC-Linux Cluster SUN Sparcstation SGI Origin 2000 IBM SP-2

9. August 2001 Intro to SMS (contd) Models Parallelized –Ocean : ROMS, HYCOM, POM –Mesoscale Weather : FSL RUC, FSL QNH, NWS Eta, Taiwan TFS (Nested) –Global Weather : Taiwan GFS (Spectral) –Atmospheric Chemistry : NOAA Aeronomy Lab

10. August 2001 Key SMS Directives Data Decomposition –csms$declare_decomp –csms$create_decomp –csms$distribute Communication –csms$exchange –csms$reduce Index Translation –csms$parallel Incremental Parallelization –csms$serial Performance Tuning –csms$flush_output Debugging Support –csms$reduce (bitwise exact) –csms$compare_var –csms$check_halo

11. August 2001 SMS Serial Code

12. August 2001 Advanced Features Nesting Incremental Parallelization Debugging Support (Run-time configurable) –CSMS$REDUCE Enables bit-wise exact reductions –CSMS$CHECK_HALO Verifies a halo region is up-to-date –CSMS$COMPARE_VAR Compare variables for simultaneous runs with different numbers of processors HYCOM 1-D decomp parallelized in 9 days

13. August 2001 Incremental Parallelization “global” “local” “global” CALL NOT_PARALLEL(...) SMS Directive: CSMS$SERIAL

14. August 2001 Advanced Features (contd) Overlapping Output with Computations (FORTRAN Style I/O only) Run-time Process Configuration –Specify number of processors per decomposed dim or number of grid points per processor 15% performance boost for HYCOM –Support for irregular grids coming soon

15. August 2001 SMS Performance (Eta) Eta model run in production at NCEP for use in National Weather Service Forecasts 16000 Lines of Code (excluding comments) 198 SMS Directives added to the code

16. August 2001 ETA Performance Performance measured on NCEP SP2 I/O excluded Resolution : 223x365x45 88 PE run-time beats NCEP hand-coded MPI by 1% 88 PE Exchange time beats hand-coded MPI by 17%

17. August 2001 SMS Performance (HYCOM) 4500 Lines of Code (excluding comments) 108 openMP directives included in the code 143 SMS Directives added to the code

18. August 2001 HYCOM Performance Performance measured on O2K Resolution : 135x256x14 Serial code runs in 136 seconds

19. August 2001 Intro to SMS (contd) –Extensive documentation available on the web –New development aided by Regression test suite Web-based bug tracking system

20. August 2001 Outline Who we are Intro to SMS Application of SMS to ROMS Ongoing Work Conclusion

21. August 2001 SMS ROMS Implementation Used awk and cpp to convert to dynamic memory; simplifying SMS parallelization Leveraged existing shared memory parallelism do I = ISTR, IEND Directives added to handle NEP scenario 13000 Lines of Code, 132 SMS directives Handled netCDF I/O with CSMS$SERIAL

22. August 2001 Results and Performance Runs and produces correct answer on all supported SMS machines Low Resolution 128x128x30 –“Jet”, O2K, T3E Scaling –Run-times for main loop (21 time steps) excluding I/O High Resolution 210x550x30 –PMEL using in production –97% Efficiency between 8 and 16 processors on “Jet”

23. August 2001 SMS Low Res ROMS “Jet” Performance

24. August 2001 SMS Low Res ROMS O2K Performance

25. August 2001 SMS Low Res ROMS T3E Performance

26. August 2001 Outline Who we are Intro to SMS Application of SMS to ROMS Ongoing Work Conclusion

27. August 2001 Ongoing Work (funding dependent) Full F90 Support Support for parallel netCDF T3E port SHMEM implementation on T3E, O2K Parallelize other ROMS scenarios Implement SMS nested ROMS Implement SMS coupled ROMS/COAMPS

28. August 2001 Conclusion SMS is a high level directive-based tool Simple single source parallelization Performance optimizations provided Strong debugging support included Performance beats hand-coded MPI SMS is performance portable

29. August 2001 Web-Site www-ad.fsl.noaa.gov/ac/sms.html