1. Computation of High-Resolution Global Ocean Model using Earth Simulator By Norikazu Nakashiki (CRIEPI) Yoshikatsu Yoshida (CRIEPI) Takaki Tsubono (CRIEPI) Dong-Hoon. Kim (CRIEPI) Frank O. Bryan (NCAR) Richard D. Smith (LANL) Mathew E. Maltrud (LANL) Julie L. McClean (NPS)

2. Parallel Ocean Program (POP) 1.Designed for Massive Parallel Computer -> sheared memory, massive parallel computing 2. Free-surface boundary condition -> no island problem -> unsmoothed bottom topography -> prognostic sea-surface height 3. General Orthogonal Coordinate -> displaced-pole grid (singularity free Arctic Ocean) 4. Vertical mixing parameterization 1) simple constant mixing 2) Richardson-number dependent mixing 3) KPP mixing parameterization

3. 5. Convective Adjustment 1) convection adjustment 2) large mixing coefficient 6. Horizontal mixing 1) laplacian 2) bi-harmonic 3) Gent-McWilliams isopycnal tracer diffusion 4) Anisotropic viscosity 7. Equation of State 1) UNESCO eq. (based on potential temperature) 2) full UNESCO eq. (polynomial fit) 3) linear eos 8. Topographic stress 1) Holloway’s topographic stress parameterization

4. POP (Parallel Ocean Program) 1)High Resolution Global Ocean model Resolution : 0.1x0.1x40L ( 3600x2400x40) (pole on North America) Horizontal : Bi-harmonic Mixing for Momentum & Tracer Vertical : Kpp Mixing Time step : 220/day ( ≒ 6min.) 2) Global Model for CCSM2 Resolution : 1x1x40L ( 320x384x40) (pole on Green Land) Horizontal : Anisotropic Mixing for Momentum GM Mixing for Tracer Vertical : Kpp Mixing Time step : 23/day ( ≒ 60min.)

5. Computational Grid of POP x0.1 Horizontal Mesh Vertical Mesh

6. POP timing measurement on ES 1 degree model –320 x 384 x 40 grid division, 23 full-step/day –KPP vertical mixing scheme –GM horizontal mixing for tracer –Anisotropic viscosity parameterization –3rd upwind tracer advection 0.1 degree model –3600 x 2400 x 40 grid division, 220 full-step/day –KPP vertical mixing scheme –Bi-harmonic horizontal mixing for tracer and momentum No history output. No forcing data input.

7. w/o optimization w/ optimization (a) 20 PEs (b) 160 PEs (c) 960 PEs baroclinic barotropic baroclinic barotropic Cost distribution in POP resolution: x0.1 deg

8. Scalability in baroclinic/barotropic mode Significant improvement in barotropic mode Scalability wall around 2-node (1 deg) and 80-node (0.1 deg) Slight speedup in baroclinic mode (a) 1 degree(b) 0.1 degree

9. POP performance on ES 1.64 day/century,70.2 Gflop/s,at 1 degree (4 nodes) 27.1 day/century,1.60 Tflop/s,at 0.1 degree (120 nodes) (a) wallclock v.s. # of PEs(b) efficiency v.s. # of PEs

10. Parallel Efficiency of POP x1 (Relation with Vertical Resolution) Parallel efficiency ≧ 50 % (10 Node ) on ES center Num. of PE Wall Clock Time (sec) for 2 Days Integration Parallel Efficiency (%) Vertical Resolution -> 40L, 80, 160, 200L

11. Further optimizations for POP code POP version 1 –Distributed parallel I/O w/ horizontal data decomposition (J. Ueno, will be completed in March) –Tests of NEC’s new MPI library (incl. all-reduce) –Merge CRIEPI version and CRAY version into one POP version 2 –POP2 beta2 code ported to ES –Vector optimization –Timing measurement in progress (H. Komatsu, J. Ueno) Some problems w/ OpenMP NEC’s compiler supports OpenMP1.1, not OpenMP2 Some features of f90 cannot be used w/ OpenMP1.1

12. 10 years Spin up of POP (x0.1) 10 year Computation (10year * 1cycle) Initial data From LANL/NPS Earth Simulator 40 node (320 PE) Atmospheric Boundary Conditions NCEP, etc. (1990-2000) (1) Wind Daily (2) Surface Heat Flux Daily (3) Surface Fresh Water Flux Monthly POP x0.1 Surface Boundary Condition

13. Global Diagnostics Kinetic Energy at Surface Global Mean KE Global Mean PTGlobal Mean SAL

14. Annual Mean Sea Surface Temp.

15. Annual Mean Sea Surface Sal. Levitus POP x1 (2000) POP x0.1 m(2000)

16. Kuroshio CCSM2 (for climate simulation) X1 deg. (100km×100km) High Resolution Model x0.1 deg. (10km× １ 0km)

17. CCSM2 (for climate simulation) X1 deg. (100km×100km) High Resolution Model x0.1 deg. (10km× １ 0km) Equatorial Current

18. Sea Surface Temp. Glonbal1990-2000 Monthly SST 1990-1991 Daily SST Kuroshio Gulf Stream 1990-2000 Monthly Vel.

19. Kuroshio Ohsumi Tsushima Tsugaru Soya Tokara SSH & Volume Transport Section

20. Izu 30-60 Sv Tokara 13 Sv Soya 0.7Sv Tsugaru 1.5Sv Tsushima 2.2Sv Japan Sea Kuroshio Volume Transport

21. Sensitivity Analysis of POP x0.1 To improve Gulf Stream & Kuroshio, etc. → Change Strength of Horizontal Mixing Viscosity & Diffusivity of Bi-harmonic Mixing case 01a: am = -2.7e18, ah = -9.0e17 Same Horizontal Mixing (basic) case 01b: x1/2 case 01c: x1/3 Surface Forcing : Monthly Climatology

22. Global Diagnostics Global Mean KE Global Mean PT Global Mean SAL case 01a Same (basic) case 01b x1/2 case 01c x1/3

23. SSH case 01a x1/3 9 th, 10 th year 9 th 10 th

24. SSH case 01c x1/3 9 th, 10 th year 9 th 10 th

25. Viscosity – SSH (Kuroshio) Global Mean KE - 01a basic - 01b x1/2 - 01c x1/3 x1/2 x1/3 basic Low High

26. Tsushim a Tsugar u Soy a Es_01a Es_01bEs_01c

27. (Sv) Tsushima Soya Tsugaru Volume Transport

28. Tokara

29. Kuroshio

30. case_01c year-12, Jan.,Mar.,and May Jan MarMay

31. New Sections are planning for Volume Transport Checking

32. case 01a Same (basic) case 01b x1/2 case 01c x1/3 Volume Transport (Kuroshio) Global Mean KE - 01a basic - 01b x1/2 - 01c x1/3

33. Viscosity – SSH (Gulf Stream) x1/2 x1/3 basic

34. Sensitivity Analysis of POP x0.1 To improve Gulf Stream, etc. → Change Restoring condition, Topography Viscosity & Diffusivity of Bi-harmonic Mixing case 01e viscosity & diffusivity x1/3 + w/o restoring case 01f viscosity & diffusivity x1/3 + w/ topography change

35. case 01f w/ topography change

36. case 01f x1/3 w/ topo. changecase 01e x1/3 w/o restoring SSH at 6 th year

37. Global Diagnostics Global Mean KE Global Mean PT Global Mean SAL case 01a Same (basic) case 01b x1/2 case 01c x1/3 case 01d GM scheme case 01e x1/3 w/o restoring case 01f x1/3 w/ topo. change

38. Global Diagnostics Global Mean KE Global Mean PT Global Mean SAL case 20d (basic) case 00a GM mixing With NCEP daily forcing

39. Future Research Plan 1)POP x0.1 deg. * Improvement of the Model Sensitivity Analysis Horizontal & Vertical Mixing etc. Vertical Resolution 40 Layer -> 106 Layer Active Ice Model 2)POP x1 deg. * Tuning for CCSM2 Computation

40. Research Plan in FY2003 1)POP x0.1 deg. * Improvement of the Model Sensitivity Analysis : Horizontal & Vertical Mixing etc. Vertical Resolution : 40 Layer -> 106 Layer Active Ice Model ? * Analysis the Results and Write Paper 2)POP x1 deg. * Tuning for CCSM2 Computation 3)Regional Nesting model Porting to ES Center Nesting to POP x1 deg.