1. Slide 1 CLOUDS AND STORMS PROJECT ATHENA Very high resolution global climate simulations with the ECMWF model Martin Miller ECMWF (Acknowledgements to many colleagues) (Sept 2011)

2. Slide 2 Outline ● Introduction: The value of very high-resolution in global NWP ● Project Athena – the value of very high resolution in climate simulations ● Future developments in the ‘Grey Zone’

3. Slide 3 The value of higher resolution  Many features of the real atmosphere are still grossly under-resolved in models (especially for ensemble systems and in severe weather conditions)  Errors from under-resolved scales affect the entire spectrum of motions  Increases in resolution lead to reduced systematic errors and better kinetic energy spectra  More observations can be used at higher resolution, and more information can be extracted from observations  Resolution increases have accounted for a significant proportion of past improvements in forecast skill CAS2K7 Lecture

4. Slide 4 Historic Evolution of Skill at ECMWF

5. Slide 5 Systematic Error: D+10 DJF Z500 Stream Function 200hPa Velocity Potential 200hPa 1986-19891996-19992006-2009

6. Slide 6 TAC 42 Verification 2010 Tropical cyclones TC predictions for 12-month periods ending on 14 July each year.

7. Weather Prediction (T1279, ~15 km) compared with Satellite Observations ECMWF predictions and Meteosat observations

9. Kinetic Energy Spectra (500hPa) T3999 and T1279 Hyd. IFS T L 3999 T L 1279

10. Slide 10 Project Athena Very High Resolution Global Climate Simulations (uncoupled) with the ECMWF Model Impact of resolution on GCM climate, variability, and phenomena including time slice experiments (Slides mostly provided by Thomas Jung) Formerly at ECMWF now at AWI - Germany

11. Slide 11 Overview  Background and goals  Experimental setup  Computational aspects  Key scientific results Model climate Phenomena Time- slice  Summary

12. Slide 12  The World Modelling Summit (WMS) in May 2008 called for a revolution in climate modelling to accelerate improvements in accuracy and reliability  The WMS recommended that peta-scale supercomputers dedicated to climate modelling be based in at least three international facilities  The U.S. National Science Foundation, recognizing the importance of the problem, offered to support the challenge of the World Modelling Summit by providing the dedicated supercomputer (Athena) at ORNL/NICS for 6 months in 2009- 2010  An international collaboration was formed among groups in the U.S., Japan and the U.K. to take up the challenge – Project Athena, a pilot project to demonstrate the power of international collaboration with dedicated computing resources Background

13. Slide 13  Hypothesis: Accurately resolving sub-synoptic and mesoscale phenomena in the atmosphere → much improved fidelity of the models in Simulating the mean climate, its variances and co-variances, (and occurrence of extreme events). Predicting climate anomalies on seasonal and longer time scales  Hypothesis: Explicitly resolving important processes such as cloud motions and transports without parametrization → improved fidelity of the models in representing important phenomena crucial for regional weather and climate.  Hypothesis: Simulating the effect of increasing greenhouse gases on regional aspects of climate, such as precipitation and storminess and their extremes, may depend critically on the spatial resolution of the global climate model. Project Athena: Science goals

14. Slide 14 COLA - Center for Ocean-Land-Atmosphere Studies, USA (NSF-funded) ECMWF - European Centre for Medium-range Weather Forecasts, UK JAMSTEC - Japan Agency for Marine-Earth Science and Technology, Research Institute for Global Change, Japan University of Tokyo, Japan NICS - National Institute for Computational Sciences, USA (NSF-funded) Cray Inc. Codes IFS: ECMWF Integrated Forecast System NICAM: Nonhydrostatic Icosahedral Atmospheric Model Supercomputers Athena: Cray XT4 - 4512 quad-core Opteron nodes (18048) 0.16 PF #30 on Top500 list (November 2009) – dedicated Oct’09 – Mar’10 Kraken: Cray XT5 - 8256 dual hex-core Opteron nodes (112896) 1.2 PF #3 on Top500 list (November 2009) replaced Athena – allocation of 5M SUs Project Athena: Overview Partners

15. Slide 15

16. Slide 16 Main IFS experiments (uncoupled) T159T511T1279T2047 Resolution (km)125401510 Radiation gridT63T159T511T639 Time step (min)6015107.5 3-month 1 2001-2009─ 13-months 2 1960-2007 1989-2007 AMIP 3 1960-2007─ ─ Time slice 4 2070-2117─ ─ Seasonal forecasts 5 Sel. cases─ ─ 1 Forecasts started on 21 May covering June-August. 2 Forecasts started on 1 November. 3 Forecast started on 1 November 1960. 4 More details below. 5 10 member lagged ensemble (see below). NICAM experiments7 km8 cases21 May-30 Aug 2001-2009

17. Slide 17 Models and Machine Usage  IFS flexible scalability sustained good performance for higher resolution configurations (T1279 and T2047) using 2,560 processor cores  We defined one “slot” as 2,560 cores and managed a mix of NICAM and IFS jobs @ 1 job per slot  optimal use of resource.  Having equal size slots for both models permits either model to be queued and run in the event of a job failure.  Selected jobs given higher priority so that they continue to run ahead of others.  Machine partition: 7 slots of 2,560 cores = 17,920 cores out of 18,048  99% machine utilization  128 processors for pre- and post-processing and as spares (postpone reboot)  Lower resolution IFS experiments (T159 and T511) were run on Kraken  IFS runs were initially made by COLA but when the ECMWF SMS model management system was installed, runs could be made by COLA or ECMWF. Courtesy of Larry Marx, COLA

18. Slide 18 Computational aspects  Dedicated machine enabled ~95% capacity to be achieved over the 6 month period (72 million core hours out of 78.8).  Several hundred simulated years for the IFS with just a few simulated years with NICAM since :  IFS timestep at 10kms =450secs NICAM timestep at 7km=30 secs (fewer assumptions) (actually about a factor of 50 more expensive) For the IFS the increased cost compared to T159:  33 times for T511  520 times for T1279  2136 times for T2047

19. Slide 19 Post-processing Almost a Petabyte of data  Focus on low-resolution data: Spectral data truncated to T159. Grid point data interpolated to N80.  Pros: Direct comparison with reanalyses. Well matched to observations. Possible to archive more parameters at higher temporal resolution.  Cons: Focus on larger synoptic and planetary scales.  Compromise: Go to T511/N256?

20. Slide 20 Total Precipitation DJF

21. Slide 21 Synoptic Activity: Z500 (JJA 1990-2008) Joint ICSC-WGNE Session, Offenbach, 3 Nov 2009 Slide 21 T159-ReanalysisT511-T159 T2047-T1279T1279-T511

22. Slide 22 Extratropical cyclones DJF

23. Slide 23 Average number of extratropical cyclones Sourceξ 850 SLP December-February ERA-Interim130.880.8 T159122.274.4 T511130.779.2 T1279128.477.3 T2047130.577.3 T159 15min 119.372.5 T511 O159 131.379.7 June-August ERA-Interim100.067.9 T15989.561.2 T51197.164.6 T127998.663.8 T204799.764.0 T159 15min 86.359.5  Northern Hemisphere  Min. lifetime ≥ 2 days  Min. migration distance ≥ 2000km  Data truncated to T159/N80

24. Slide 24 Extratropical cyclones Vorticity 850mb Wind speed 925mb

25. Slide 25 Flow across Greenland and resolution T95 T255 T799 Jung and Rhines, JAS, 2007 Sensible plus latent heat fluxes

26. Slide 26 N. ATL Hurricanes

27. Slide 27 TRMM JJA (’98-’09) Diurnal cycle

28. Slide 28 ‘AMIP’ Euro-Atlantic blocking

29. Slide 29 Euro-Atlantic blocking ‘AMIP’

30. Slide 30 The Quasi-Biennial Oscillation Joint ICSC-WGNE Session, Offenbach, 3 Nov 2009 QBO period depends on GW activity Too much total GW activity in T1279? Reduction of parameterized GW amplitude Figure courtesy of Peter Bechtold

31. Slide 31 Zonal Mean Temperature (DJF) Joint ICSC-WGNE Session, Offenbach, 3 Nov 2009 Slide 31 T159-ERA40 T511-T159 T1279-T511T2047-T1279

32. Slide 32 Time slice experiments  Initial conditions from 1 November 1970.  Modification of SST and sea ice fields: Added perturbations. Difference between 2065-2075 and 1965-1975 monthly climatologies. SST Data from Community Climate System Model run.  Greenhouse gas concentrations follow A1B until 2100 (constant thereafter).

33. Slide 33 Precipitation Change (JJA 21 st C – 20 th C) Time Slice-AMIP (T159)Time Slice-AMIP (T1279) Time Slice-AMIP (T1279-T159)

34. Slide 34 Euro-Atlantic blocking: Time slice

35. Slide 35 Precipitation Change Europe: Apr-Oct 21 st C minus 20 th C T159 (125-km)T1279 (16-km) “Time-slice” runs of the ECMWF IFS global atmospheric model with observed SST for the 20 th century and CMIP3 projections of SST for the 21 st century at two different model resolutions. The continental-scale pattern of precipitation change in April – October (growing season) associated with global warming is similar, but the regional details are quite different, particularly in southern Europe.

36. Slide 36 Seasonal forecasts: Case selection SeasonYearClimate anomaly DJF2005/06Euro-Atlantic blocking DJF1997/98Largest El Nino DJF1998/99Largest La Nina DJF1982/83Large El Nino DJF1988/89Large La Nina DJF1976/77Extreme North American weather JJA1988Record drought in North America and India JJA1993Record flooding in North America JJA2003Record heat wave in Europe JJA2005Record Atlantic hurricane season JJA1997Early onset El Nino with teleconnections JJA2004Active typhoon season in western Pacific

37. Slide 37 Winter 2005/06: Z500 Anomalies Joint ICSC-WGNE Session, Offenbach, 3 Nov 2009 Slide 37 ReanalysisT159 T511T1279 Cold European winter Tropical Relaxation

38. Slide 38 Winter 1997/98: Z500 Anomalies Joint ICSC-WGNE Session, Offenbach, 3 Nov 2009 Slide 38 ReanalysisT159 T511T1279 Largest El Nino

39. Slide 39 Closing remarks /1  From a computational viewpoint - Project Athena has been a real success not least in demonstrating the value of dedicated computing  The experience suggests that there would be significant differences between a dedicated facility for a small number of large projects versus the more usual large number of diverse users  An expert international team including computational experts on-site with dedicated facilities led to rapid solutions to issues in real time and an experimental achievement in 6 months that would have taken several years otherwise  Over 1 PB of data needs to be re-archived and/or reduced

40. Slide 40 Closing remarks /2  Model climate (sensitivity to resolution): Some improvements  Euro-Atlantic blocking (orography).  Level of synoptic activity (storm tracks etc)  Some aspects of tropical precipitation  Near-surface winds in the western tropical Pacific  Achieved mostly by going from T159 to T511!  Tropical cyclone strength Some neutral (eg MJO and Indian Summer Monsoon) Some deteriorations (eg QBO and stratospheric temps.)  Time slice experiments Large-scale response similar but large regional changes.

41. Slide 41 Closing remarks /3  Much more diagnosis is needed and is in progress including several interesting comparisons between the IFS (parametrized convection) and the NICAM (explicit convection) and the issues thereby  Repeat with coupled system (now possible?)  Several publications are already published or in press including: Project Athena itself (Kinter et al) The simulated climate and sensitivity to resolution (Jung et al) The hydrological cycle (Dirmeyer et al) The MJO and tropical cyclones etc (Satoh et al)