1. Comparing and Contrasting Post-processing Approaches to Calibrating Ensemble Wind and Temperature Forecasts Tom Hopson Luca Delle Monache, Yubao Liu, Gregory Roux, Wanli Wu, Will Cheng, Jason Knievel, Sue Haupt

2. Army Test and Evaluation Command: Dugway Proving Ground

3. Dugway Proving Grounds, Utah e.g. T Thresholds Includes random and systematic differences between members. Not an actual chance of exceedance unless calibrated.

4. Xcel Energy Service Areas Wind Farms (50+) ~3200 MW Northern States Power (NSP) Public Service of Colorado (PSCO) Southwestern Public Service (SPS) 3.4 million customers (electric) Annual revenue $11B Copyright 2010 University Corporation for Atmospheric Research

5. WRF RTFDDA Model Domains Ensemble System (30 members) D1 = 30 km D2 = 10 km 0-48 hrs Real Time Four Dimensional Data Assimilation (RTFDDA) 41 vertical levels Vary: Multi-models Lateral B.Cs. Model Physics External forcing Yubao Liu -- yliu@ucar.edu for further questionsyliu@ucar.edu

6. Goals of an EPS Predict the observed distribution of events and atmospheric states Predict uncertainty in the day’s prediction Predict the extreme events that are possible on a particular day Provide a range of possible scenarios for a particular forecast

7. Outline I.Brief overview of: 1) quantile regression (QR), 2) logistic regression (LR), 3) umbrella post- processing procedure, 4) “analog Kalman filter” (ANKF) II.2 nd moment calibration via rank histograms III.Skill score comparisons and improvements with increased hindcast data III.Example of blending approaches IV.Conclusions

8. Example of Quantile Regression (QR) Our application Fitting T quantiles using QR conditioned on: 1)Ranked forecast ens 2)ensemble mean 3)ensemble median 4) ensemble stdev 5) Persistence Hopson and Hacker 2012

9. Logistic Regression for probability of exceedance (climatological thresholds)

10. Probability/°K Temperature [K] Probability/°K Temperature [K] Forecast PDF climatological PDF Step I: determine climatological quantiles Probability Temperature [K] Step 3: use conditioned CDF to interpolate desired quantiles 1.0 0.5.75.25 prior posterior Step 2: calculate conditional probs for each climat quan Probability Temperature [K] 0.5.75.25 1.0 Final result: “sharper” posterior PDF represented by interpolated quans Hopson and Hacker 2012

11. T [K] Time forecastsobserved Regressor set: 1. reforecast ens 2. ens mean 3. ens stdev 4. persistence 5. LR quantile (not shown) Probability/°K Temperature [K] climatological PDF Step I: Determine climatological quantiles Step 2: For each quan, use forward step-wise cross-validation to select best regress set Selection requires: a) min QR cost function, b) binomial distrib at 95% confidence If requirements not met, retain climatological “prior” 1. 3. 2. 4. Step 3: segregate forecasts based on ens dispersion; refit models (Step 2) for each range Time forecasts T [K] I.II.III.II.I. Probability/°K Temperature [K] Forecast PDF prior posterior Final result: “sharper” posterior PDF represented by interpolated quans Hopson and Hacker 2012

12. National Security Applications Program Research Applications Laboratory Significant calibration regressors 3hr Lead-time 42hr Lead-time Station DPG S01

13. National Security Applications Program Research Applications Laboratory RMSE of ensemble members 3hr Lead-time 42hr Lead-time Station DPG S01

14. Time t = 0 day -1 day -2 day -6 day -5 day -4 day -3 day -7 OBS PRED KF-weight KF Delle Monache et al. 2010 Analog Kalman Filter (ANKF) Deterministic method applied to each individual ensemble KF weighting run in analog space

15. Time t = 0 day -1 day -2 day -6 day -5 day -4 day -3 day -7 OBS PRED KF-weight KF ANKF AN “Analog” Space day -4 day -7 day -5 day -3 day -2 day -1 day -6 PRED OBS farthest analog closest analog NOTE This procedure is applied independently at each observation location and for a given forecast time Delle Monache et al. 2010

16. Outline I.Brief overview of: 1) quantile regression (QR), 2) logistic regression (LR), 3) umbrella post- processing procedure, 4) “analog Kalman filter” (ANKF) II.2 nd moment calibration via rank histograms III.Skill score comparisons and improvements with increased hindcast data III.Example of blending approaches IV.Conclusions

17. 42-hr dewpoint time series Before Calibration After Calibration (QR) Station DPG S01

18. Original ensemble QR LRANKF Rank Histograms 15hr lead wind forecasts

19. Skill measures used: 1)Rank histogram (converted to scalar measure) 2)Root Mean square error (RMSE) 3)Rank Probability Score (RPS) 4)Relative Operating Characteristic (ROC) curve Skill Scores Comparing to original ensemble forecast, but with bias removed => “reference forecast”

20. Blue - QR Red - ANKF Green - LR Skill Score Comparison For wind farm CEDC, 3hr lead forecasts Reference forecast: original wind speed ensemble w/ bias removed Data size: 900pts

21. Rank Histogram scalar QR LR RMSE Skill Scores Dependence on Training Data Size Upper dashed – 900pts Solid – 600pts Lower dashed – 300pts Reference Forecast: Original wind speed ensemble w/ bias removed

22. ROC QR LR RPSS Skill Scores Dependence on Training Data Size (cont) Reference Forecast: Original wind speed ensemble w/ bias removed Upper dashed – 900pts Solid – 600pts Lower dashed – 300pts

23. 23 RPS ROC RMSE Brier Score Wind farm TWBT 1 1 1 1 12 24 36 48 36 ANKF QR QR + ANKF

24. 24 original ANKF QR QR + ANKF TWBT 6-h

25. Summary  “step-wise cross-validation”-based post-processing framework provides a method to ensure forecast skill no worse than climatological and persistence  Also provides an umbrella to blend together multiple post-processing approaches as well as multiple regressors, and to diagnose their utility for a variety of cost functions  Quantile regression and logistic regression useful tools for improving 2 nd moment of ensemble distributions  See significant skill gains with increasing “hindcast data” amount for a variety of skill measures  Blending of post-processing approaches can also further enhance final forecast skill (e.g. ANKF and QR) by capturing “best of both worlds” Further questions: hopson@ucar.edu or yliu@ucar.eduhopson@ucar.eduyliu@ucar.edu