1. The University of Washington Mesoscale Short-Range Ensemble System Eric P. Grimit, F. Anthony Eckel, Richard Steed, Clifford F. Mass University of Washington

2. The UW Mesoscale Ensemble System: The Big Picture The UW Mesoscale Ensemble System was born out of our experience with high-resolution prediction: MM5 run at 36, 12 and 4 km twice a day for many years. High-resolution forecasts can produce highly realistic mesoscale structures, but there is considerable uncertainty in initial conditions and physics. High resolution can amplify such uncertainty and thus it is dangerous to provide users with high resolution output for direct and literal use. Mesoscale ensembles are probably the best way to provide the probabilistic information required by users…information they are currently denied…but there are significant roadblocks that need to be overcome.

3. Mesoscale Ensembles: In its Infancy At a national level, mesoscale ensembles are at a very primitive stage: NCEP’s system at 48 km grid spacing is not really on the mesoscale and uses a method (breeding) that is probably not ideal for short range ensembles. Operational SREF have not had bias removal or proper post-processing There have been a few short-term ensemble experiments (e.g. SAMEX)--generally for convection The value of mesoscale SRRF have not been proven, useful intuitive products are lacking, and there is little experience in the user community. But most of us are convinced that this is the way to go.

4. The UW Mesoscale Ensemble System Essential Features A true mesoscale system: 36 - 12 km. Out to 48 h Testing the value of mesoscale ensembles over a different environment: eastern Pacific, coastal zone, area of terrain. Moist to desert locations. The diversity generation is based on using the varying initial conditions and boundary conditions from a broad range of operational synoptic models… all with differing data assimilation, model structure and numerics, and physics. Finesses BC problem. This approach is politically unacceptable to many operational centers (who don’t like to be dependent on others), but probably represents a high bar for others to attempt to better. Additional diversity from varying model physics and surface boundary conditions.

5. UW Mesoscale Ensemble System Single limited-area mesoscale modeling system (MM5) 2-day (48-hr) forecasts at 0000 UTC in real-time since January 2000. Now twice a day 36 and 12-km domains. Configurations of the MM5 short-range ensemble grid domains. (a) Outer 151  127 domain with 36-km horizontal grid spacing. (b) Inner 103  100 domain with 12-km horizontal grid spacing. a)b) 36-km 12-km

6. Resolution ( ~ @ 45  N ) Objective Abbreviation/Model/Source Type Computational Distributed Analysis avn, Global Forecast System (GFS), SpectralT254 / L641.0  / L14 SSI National Centers for Environmental Prediction~55 km~80 km3D Var cmcg, Global Environmental Multi-scale (GEM), Finite0.9  0.9  /L281.25  / L113D Var Canadian Meteorological Centre Diff ~70 km ~100 km eta, limited-area mesoscale model, Finite32 km / L45 90 km / L37SSI National Centers for Environmental Prediction Diff.3D Var gasp, Global AnalysiS and Prediction model,SpectralT239 / L291.0  / L11 3D Var Australian Bureau of Meteorology~60 km~80 km jma, Global Spectral Model (GSM),SpectralT106 / L211.25  / L13OI Japan Meteorological Agency~135 km~100 km ngps, Navy Operational Global Atmos. Pred. System,SpectralT239 / L301.0  / L14OI Fleet Numerical Meteorological & Oceanographic Cntr. ~60 km~80 km tcwb, Global Forecast System,SpectralT79 / L181.0  / L11 OI Taiwan Central Weather Bureau~180 km~80 km ukmo, Unified Model, Finite5/6  5/9  /L30same / L123D Var United Kingdom Meteorological Office Diff.~60 km “Native” Models/Analyses Available

7. UW Ensemble System Made use of the infrastructure already in place (grids, data feeds, systems and application programmers), plus took advantage the natural parallelization using large clusters..which are ideal for ensemble work. The system was built and maintained by two exceptional graduate students (Eric Grimit and Tony Eckel), plus key staff members (Rick Steed, David Ovens) Was designed as a real-time system from the beginning, with verification as a core component Had two operational groups as prime subjects: the Seattle NWS office and the Navy Whidbey Island forecasting detachment. Had strong partners with UW Statistics and APL (under MURI support)

8. “Ensemblers” Eric Grimit (l ) and Tony Eckel (r) are besides themselves over the acquisition of the new 20 processor Athlon cluster Computer Infrastructure: Linux Dual- Processor Clusters

9. Key Goals: End-to-End Evaluation To build a viable, operational mesoscale SREF To verify it using both deterministic (ensemble mean) and probabilistic approaches. To determine whether a system with members of varying skill can be combined to produce reliable and usefully sharp ensemble pdfs. To determine the best approaches for post-processing (e.g., bias removal, calibration, optimal pdf generation) To determine whether the ensemble system can be used to predict deterministic and probabilistic skill To create ensemble-based products that are valuable to users. To learn how to optimally combine high resolution deterministic forecasts and lower-res ensembles

10. UW Ensemble Web Page

13. 48 h Probabilistic Forecast Of 1 inch In 12h:

14. UW’s Ensemble of Ensembles # of EF Initial Forecast Forecast Name Members Type Conditions Model(s) Cycle Domain ACME 17SMMA 8 Ind. Analyses, “Standard” 00Z 36km, 12km 1 Centroid, MM5 8 Mirrors UWME 8SMMA 8 Independent “Standard” 00Z 36km, 12km Analyses MM5 UWME+ 8PMMA 8 Independent 8 MM5 00Z 36km, 12km Analyses variations 8 PME 8 MMMA8 Independent operational, 00Z, 12Z 36km Analyses large-scale Homegrown Imported ACME: Analysis-Centroid Mirroring Ensemble PME: Poor Man’s Ensemble MM5: 5 th Generation PSU/NCAR Mesoscale Modeling System SMMA: Single Model Multi-Analysis PMMA: Perturbed-model Multi-Analysis MMMA: Multi-model Multi-Analysis

15. Multi-Analysis, Mixed Physics : UWME + see Eckel (2003) for further details

16.  Total of 129, 48-h forecasts (Oct 31, 2002 – Mar 28, 2003) all initialized at 00z - Missing forecast case days are shaded  Analyzed Parameters :  - 36 km Domain: Mean Sea Level Pressure (MSLP), 500mb Geopotential Height (Z 500 - 12 km Domain: 10-m Wind Speed (WS 10 ), 2-m Temperature (T 2 ) Research Dataset 36 km Domain (151  127) 12 km Domain (101  103)  Verification: - 36 km Domain: centroid analysis (mean of 8 independent analyses, available at 12-h increments) - 12 km Domain: RUC20 analysis (NCEP 20 km mesoscale analysis, available at 3-h increments) NovemberDecemberJanuary February March

17. Subjective Evaluation Often large differences in initializations and forecasts Very useful forecasting tool

18. Thanksgiving Day 2001 Wind Forecast Bust eta-MM5 Initialized 00z, 21 Nov 01 (Tue. evening) 42h Forecast, valid 10AM Thursday Eta-MM5 model 12-km runs on Tue & Wed forecast severe wind storm for the Puget Sound on Thu AM. Expected widespread damage and power outage was all over the news. Verification, 10AM Thursday The storm came ashore weaker and further south giving light and variable winds in the Puget Sound.

19. 42h forecast (valid Thu 10AM) 13: avn* 11: ngps* 12: cmcg* 10: tcwb* 9: ukmo* 8: eta* Verification 1: cent 7: avn 5: ngps 6: cmcg 4: tcwb 3: ukmo 2: eta - Reveals high uncertainty in storm track and intensity - Indicates low probability of Puget Sound wind event SLP and winds

20. The Importance of Grid-Based Bias Removal Particularly important for mesoscale SREF in which model biases are often large Significantly improves SREF utility by correctly adjusting the forecast PDF

21. Training Period Bias-corrected Forecast Period Training Period Bias-corrected Forecast Period Training Period Bias-corrected Forecast Period Gridded Bias Removal N number of forecast cases (14) f i,j,t forecast at grid point (i, j ) and lead time (t ) o i,j observation (centroid-analysis or ruc20 verification) For the current forecast cycle: 1) Calculate bias at every grid point and lead time using previous 2 weeks’ forecasts 2) Postprocess current forecast to correct for bias: f i,j,t bias-corrected forecast at grid point (i, j ) and lead time (t) * NovemberDecemberJanuary February March

22. Average RMSE (  C) and (shaded) Average Bias Uncorrected ACME core+ T 2 12 h 24 h 36 h 48 h

23. Average RMSE (  C) and (shaded) Average Bias Bias-Corrected ACME core+ T 2 12 h 24 h 36 h 48 h

24. Physics and Surface Diversity Substantially Enhance a Mesoscale SREF, Particularly for Surface Quantities

25. (d) T 2 (c) WS 10 (b) MSLP (a) Z 500 *UWME *UWME+ VOP 5.0 % 4.2 % 9.0 % 6.7 % 25.6 % 13.3 % 43.7 % 21.0 % Surface/Mesoscale Variable ( Errors Depend on Model Uncertainty ) Synoptic Variable ( Errors Depend on Analysis Uncertainty ) Comparison of 36-h VRHs *UWME *UWME+ *UWME *UWME+ *UWME *UWME+

26. *UWME core UWME core *UWME core+ UWME core+ Uncertainty Skill for P(T 2 < 0°C) *Indicates bias removal Importance Of Bias Removal And Physics Diversity

27. Smaller Scales Generate Ensemble Dispersion Ensemble Variance (m 2 /s 2 ) UWME core WS 10 36-km 12-km 36­km Grid Spacing 12­km Grid Spacing 3­h Cumulative Precipitation

28. Using members with varying skill is OK, but there is a limit to how bad a member can be and still add value to the ensemble. Removing Very Unskillful Members Can Help

29. Relating Forecast Skill and Model Spread Mean Absolute Error of Wind Direction is Far Less When Spread is Low

30. [c.f. Grimit and Mass 2002] UW MM5 SREF 10-m Wind Direction

31. 12-km T 2 Spatial Distribution of Local Spread-Error Correlation Domain-Averaged STD-AEM correlation ~ 0.62 Maximum Local STD-AEM correlation ~ 0.54 (no bias correction) UWME

32. A Simple Stochastic Model of Spread-Skill An extension of the Houtekamer (1993) model of spread-skill PURPOSES: 1)To establish practical limits of forecast error predictability that could be expected given perfect ensemble forecasts of finite size. 2)To address the user-dependent nature of forecast error estimation by employing a variety of predictors and error metrics. 3)To extend spread-skill analysis to a probabilistic framework of forecast error prediction.

33. A Simple Stochastic Model of Spread-Skill Statistical ensemble forecasts at a single, arbitrary location 10 4 realizations (cases) Assumed: –Gaussian statistics –statistically consistent (perfectly reliable) ensemble forecasts Varied: –temporal spread variability (  ) –finite ensemble size (M) –spread and skill metrics (continuous and categorical) 1.Draw today’s “forecast uncertainty” from a log-normal distribution (Houtekamer 1993 model). ln(  ) ~ N( ln(  f ),    2.Create synthetic ensemble forecasts by drawing M values from the “true” distribution. F i ~ N( Z,   ) ; i = 1,2,…,M 3.Draw the verifying observation from the same “true” distribution (statistical consistency). V ~ N( Z,   )

34. Idealized Spread-Error Correlations STD-AEM correlation spread STD =Standard Deviation error AEM = Absolute Error of the ensemble Mean STD-error correlation N = 10000  = 0.5 error AES = Absolute Error of a Single ensemble member AAE = ensemble-Average Absolute Error RASE = square Root of ensemble-Average Squared Error CRPS = Continuous Ranked Probability Score

35. The Conditional Error Climatology (CEC) Method Use historical errors, conditioned by spread category, as probabilistic forecast error predictions 1 2 3 4 5 Idealized, statistical ensemble forecasts. N = 2000 M = 50;  = 0.5

36. Probabilistic Forecast Error Predictability Or might use the ensemble variance directly to get a probabilistic error forecast ENS-PDF –Most skillful approach if PDF is well-forecast Idealized, statistical ensemble forecasts. N = 10000 M = 50;  = 0.5

37. Effect of Post-Processing Bias correction reduces spread-error correlations and effectiveness of the VAR-CEC approach ENS-PDF closes the gap in performance, but is still below the baseline UWME+ (14-day grid point bias correction) 12-km T 2

38. Future UW Ensemble Work Evaluation of value of temporal ensembles for adding to diversity of on-time ensembles and for prediction of ensemble skill Perfect grid-based bias removal of component members Replace physics ensemble with one based on key uncertainties in parameterizations. (Stochastic physics is not ready for prime time yet) Creation of a new generation of products to help break the ice with forecasters. Work with statistics on emos and BMA front-end to our ensemble system

39. Future Work Add model diversity using available WRF dynamical cores. Creation of web interface that combines ensemble and high-resolution products. Evaluation of value of nudging outer domain towards parent forecasts to improve diversity.

40. The END

41. Ensemble Post-Processing In several of the talks today you will be viewing some results of gridded bias correction of the individual ensemble forecasts. More terminology: * indicates bias removal All mesoscale modeling system have significant systematic biases. The biases vary by ensemble member, season, time of day, etc. Removal of such biases has a very beneficial effect on the value of ensembles.

42. Data Info Average of 65 forecasts (25 Nov 02 – 01 Feb 03) 36km domain from Rockies to central Pacific 2-week bias training for each forecast Verification: centroid analysis Results

44. Idealized Probabilistic Error Forecast Skill May use the ensemble variance directly to get a probabilistic error forecast ENS-PDF –Most skillful approach if PDF is well-forecast Predictability highest for extreme spread cases –Reinforces earlier results Continuous case Idealized, statistical ensemble forecasts. N = 10000 M = 50;  = 0.5

45. Idealized Probabilistic Error Forecast Skill Idealized, statistical ensemble forecasts. N = 10000 M = 50;  = 0.5 (categorical case)

46. The Future is Probabilistic We never will know exactly what the forecast will be due to initialization uncertainty, inadequate model physics, and other reasons. Thus, probabilistic forecasting is the only rational way to forecast. We will also gain some ability to forecast (probabilistically) forecast skill We have to retrain ourselves AND our users. The UW system is an attempt to develop and evaluate this approach using ensembles. We acutely need feedback from forecasters.

48. Probabilistic Products Currently Using Uniform Ranks (UR) method. Democratic voting (DV) method was good as MOS. UR is even better. Calibration would provide further improvements.

49. Comparison of Brier skill scores for NGM MOS and 12-km ACME core forecasts of 12h probability of precipitation accumulations greater than 0.01 in (CAT1). The skill scores are relative to the sample climatology during the period from 1 Nov 2002 – 20 Jan 2003.