Evaluation of a Mesoscale Short-Range Ensemble Forecasting System over the Northeast United States Matt Jones & Brian A. Colle NROW, 2004 Institute for Terrestrial and Planetary Atmospheres Stony Brook University Stony Brook, New York

OUTLINE Verification Method - Northeast - Seasonal view - Multiple parameters Results Conclusions

Verification Method SUMMER = May – September 2003 WINTER = October 2003 – March 2004 Scalar Measures: Contingency-based Measures: Prob.-based Measures:

SUMMER MESUMMER MAE 2mT 2mRH SLP 10mWS 10mWD 2mT 2mRH SLP 10mWS 10mWD night day night day

Near-Surface T Lowest level cloud water (~3K ft.) Warm Cool Moist Dry Example of PHYS-member spread – Eta-PBL 2mT

WINTER MEWINTER MAE NCEP BREDS GFS 2mT 2mRH SLP 10mWS 10mWD 2mT 2mRH SLP 10mWS 10mWD night day night day

21zEta-121zEta-2 21zEta+1 21zEta+2 21zEta-CTL 00zEta 00zGFS IC MEAN PHYS MEAN L 992 L f48

SUMMER MAE 2mT 2mRH SLP 10mWS 10mWD night day night day 2mT 2mRH SLP 10mWS 10mWD night day night day 0000UTC Eta 0000UTC ensemble mean 0000UTC 4-km MM5 1200UTC 4-km MM5 0000UTC ensemble mean Can the ensemble-mean beat 4km MM5 and Eta determinitistic forecasts?

WINTER MAE Can the ensemble-mean beat 4km MM5 and Eta determinitistic forecasts? 2mT 2mRH SLP 10mWS 10mWD night day night day 2mT 2mRH SLP 10mWS 10mWD 0000UTC 4-km MM5 1200UTC 4-km MM5 0000UTC ensemble mean 0000UTC Eta 0000UTC ensemble mean

SUMMER 24HP BIAS WINTER 24HP BIAS SUMMER 24HP ETS WINTER 24HP ETS Better Worse Over Pred. Under Pred. PHYS IC ALL

Verification Rank Histogram All solutions of ensemble should be equally likely. Observation should appear no different than any ensemble member. Not a measure of skill; a necessary, but not sufficient condition for a good ensemble. Perfect “flat” “U-shaped”“N-shaped”“L-shaped” Under-dispersed Over-Dispersed Biased

SUMMER RANK HIST / MISSING RATES WINTER RANK HIST / MISSING RATES 2mT 2mRH SLP 10mWS

SUMMER MAE-VAR WINTER MAE-VAR 2mT 2mRH SLP 10mWS 10mWD Corr. Coeff. MAE VAR MAE VAR C.C.

Probabilistic Precipitation Brier Score: REL = Reliability RES = Resolution (event discrimination) UNC = Uncertainty (dependent only on obs.) f i = forecast probability o i = observed probability (=1 for occurrence, =0 for non- occurrence) N t = number of forecast/event pairs for threshold, t m = number of ensemble members (m+1 probability categories) Skill Perfect Reliability No skill No resolution

SUMMER 24HP Reliability Diagrams PHYS IC ALL Sample SUMMER 24h MPC

WINTER 24HP Reliability Diagrams PHYS IC ALL Sample WINTER 24h MPC

Ensemble Post-processing Due to model imperfections, significant bias is retained even after ensemble averaging. Day-15 Day-14 Day-13 Day-12 Day-11 Day-10 Day-9 Day-8TODAY Day-7 Day-6 Day-5 Day-4 Day-3 Day-2 Day-1 Use previous 14 complete forecasts to correct forecasts starting 0000UTC today

SUMMER MISSING RATE IMPROVEMENTWINTER MISSING RATE IMPROVEMENT 2mT 2mRH SLP 10mWS Uncalibrated Calibrated

The ensemble-mean is more skillful than component members on average for daytime 2mT/10mWS, SLP, and 10mWD. Persistent biases among component members reduce the skill advantage of the ensemble-mean during other periods (e.g. nighttime 2mT/10mWS). The ensemble-mean can outperform the deterministic Eta model, and can equal the skill of a high-resolution deterministic MM5 initialized 12 hours later. The PHYS ensemble is more beneficial for forecasting surface parameters during the warm season due to greater variation among component members. The GFS initial condition leads to a superior SLP forecast compared to the poorly skilled NCEP Eta-bred members, especially during the cool season. The GFS member outperforms the ensemble-mean for SLP and 10mWD in the cool season. The ensemble has some ability to predict forecast skill and estimate the uncertainty of a forecast through ensemble spread-error correlation, especially for 10mWD. Persistent biases among component members and ensemble underdispersion for other surface parameters reduce the spread-error correlation (e.g. 2mT, 10mWS). Conclusions (1)

In warm season, low POPs have reliability for low threshold precip. events. High POPs have reliability for all thresholds. In cool season, low POPs have poor reliability for all precip. event thresholds. High POPs have reliability all precip. event thresholds. The PHYS (IC) ensemble is more skillful in POPs during the warm (cool) season. In the warm season, the Hybrid ensemble has the greatest POP skill. A 14-day bias calibration can reduce much of the bias for most parameters, improving ensemble MRs. Conclusions (2)

18-mbr Ens output Ensemble Stats Ensemble Verif. REALTIME SBU-SREF PRODUCTS

Acknowledgments ● Eric Grimit – University of Washington ● NWS – OKX ● ITPA – SBU Website ● Publication ● Jones, M.S., and B. A. Colle, 2004: Evaluation of a mesoscale short-range ensemble forecasting system over the Northeast United States. Wea. Forecasting, in preparation.

OUTLINE Verification Method Results Conclusions Future Work

Investigate for which synoptic regimes ensemble variance is most/least useful. Investigate for which synoptic regimes a post-processing technique is most beneficial (MOS vs. historical bias calibration). Reduce the inequality of skill among members by removing poorly- performing members / replacing with multiple models, multiple analysis initial conditions. Investigate alternative ensemble quantities (trimmed mean/variance, modal quantile value). Continue efforts in improving presentation of forecast uncertainty/ensemble confidence. Future Work

Verification Rank Histogram ● All solutions of ensemble should be equally likely. ● Observation should appear no different than any ensemble member. ● Not a measure of skill; a necessary, but not sufficient condition for a good ensemble. MR = “Missing Rate” Perfect “flat” “U-shaped”“N-shaped”“L-shaped” Under-dispersed Over-Dispersed Biased

Usability of Ensemble Variance ● The variance of a properly dispersed ensemble is a good representation of forecast uncertainty. ● Ensemble variance should be correlated with ensemble error, leading to an ability of the ensemble to predict ensemble skill (Houtekamer 1993). High skill Low spread Low skill High spread

Ensemble Probability Forecasts ● An ensemble distribution should present what is most probable and what is least probable, reducing the “element of surprise” (Brooks and Doswell 1993).

SUMMER MAE REDUCTIONWINTER MAE REDUCTION PHYS IC ALL

SUMMER % BEST SUMMER % WORST

2mT 24HP CASE Near-Surface T Lowest level cloud water (~3K ft.) Composite “moist” caseComposite “dry” case case KF2 KF BM GR Warm Cool Moist Dry