Patrick Tewson University of Washington, Applied Physics Laboratory Local Bayesian Model Averaging for the UW ProbCast Eric P. Grimit, Jeffrey Baars, Clifford F. Mass University of Washington, Atmospheric Sciences Research supported by: Office of Naval Research Multi-Disciplinary University Research Initiative (MURI)

4 May :50 AMSpring MURI Meeting; Seattle, WA Motivation “As high as 81! Hey, Eric! Those intervals are too wide! And 15% chance of precip? Hmm…”

4 May :50 AMSpring MURI Meeting; Seattle, WA Summary from Last Fall Mean error climatology (MEC): Ensemble-mean + its error variance over some history. Good benchmark to evaluate competing calibration methods. Generally beats the raw ensemble, even though it is not a state- dependent forecast of uncertainty. Local Bayesian model averaging (Local-BMA): Model forecast performance varies locally: BMA parameters should depend on grid point location. Train BMA using elevation, land-use, and proximity constraints. Can consistently beat MEC in tests with grid-based verification.

4 May :50 AMSpring MURI Meeting; Seattle, WA Global-BMA Calibration and Sharpness Global- BMA MEC sharpness MEC BMA FIT calibration Probability integral transform (PIT) histograms  an analog of verification rank histograms for continuous forecasts [00 UTC Cycle; October 2002 – March 2004; 361 cases]

4 May :50 AMSpring MURI Meeting; Seattle, WA Local- BMA Local-BMA Calibration and Sharpness MEC BMA FIT calibration sharpness Probability integral transform (PIT) histograms  an analog of verification rank histograms for continuous forecasts [00 UTC Cycle; October 2002 – March 2004; 361 cases] MEC

4 May :50 AMSpring MURI Meeting; Seattle, WA BMA Forecast Skill Comparison Local-BMA CRPS % improvement over MEC Global-BMA CRPS % improvement over MEC

An Observation-Based Approach to Local-BMA Development and testing: Winter-Spring 2006 Expect it to drive the MURI “killer application”  UW ProbCast. Several “tuning” parameters available, which can hopefully be optimized. Deploy it initially for MAXT2 and MINT2 forecasts. Application to mixed discrete-continuous quantities (e.g., QPF) and 2-D quantities (wind) will require further exploration.

4 May :50 AMSpring MURI Meeting; Seattle, WA An Observation-Based Approach to Local-BMA Allow BMA parameters to vary by grid point. Use observations, remote if necessary, as training data. Follow the Baars et al. procedure for bias correction (optimized from the Mass-Wedam-Steed method) to also select the training data for Local-BMA. For each grid point, search for n (e.g. 8) nearby stations (e.g. within 864-km) at similar elevation (e.g. within 250-m) and having similar land-use. Land-use categories were concatenated into 9 categories (down from 24 in MM5). Figure shows methodology for Mass-Wedam-Steed settings.

4 May :50 AMSpring MURI Meeting; Seattle, WA Maximum 2-m Temperature – Case Study Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, South-north grid point: West-east grid point : obs: F model#:1, forecast: F model#:2, forecast: F model#:3, forecast: F model#:4, forecast: F model#:5, forecast: F model#:6, forecast: F model#:7, forecast: F model#:8, forecast: F ENS-MEAN: F -----

4 May :50 AMSpring MURI Meeting; Seattle, WA Global-BMA Mean Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, South-north grid point: West-east grid point : obs: F model#:1, forecast: F model#:2, forecast: F model#:3, forecast: F model#:4, forecast: F model#:5, forecast: F model#:6, forecast: F model#:7, forecast: F model#:8, forecast: F Global-BMA-MEAN: F -----

4 May :50 AMSpring MURI Meeting; Seattle, WA Local-BMA Mean Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, South-north grid point: West-east grid point : obs: F model#:1, forecast: F model#:2, forecast: F model#:3, forecast: F model#:4, forecast: F model#:5, forecast: F model#:6, forecast: F model#:7, forecast: F model#:8, forecast: F Local-BMA-MEAN: F -----

4 May :50 AMSpring MURI Meeting; Seattle, WA Bias-Corrected Ensemble Mean Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, South-north grid point: West-east grid point : obs: F model#:1, forecast: F model#:2, forecast: F model#:3, forecast: F model#:4, forecast: F model#:5, forecast: F model#:6, forecast: F model#:7, forecast: F model#:8, forecast: F BC-ENS-MEAN: F -----

4 May :50 AMSpring MURI Meeting; Seattle, WA Global-BMA Sharpness Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, South-north grid point: West-east grid point : obs: F Global-BMA-95%: F Global-BMA-MEAN: F Global-BMA- 5%: F -----

4 May :50 AMSpring MURI Meeting; Seattle, WA Local-BMA Sharpness Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, South-north grid point: West-east grid point : obs: F Local-BMA-95%: F Local-BMA-MEAN: F Local-BMA- 5%: F -----

4 May :50 AMSpring MURI Meeting; Seattle, WA Local-MEC Sharpness Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, South-north grid point: West-east grid point : obs: F Local-MEC-95%: F Local-MEC-MEAN: F Local-MEC- 5%: F -----

4 May :50 AMSpring MURI Meeting; Seattle, WA Calibration (all stations)

4 May :50 AMSpring MURI Meeting; Seattle, WA Calibration (water only)

4 May :50 AMSpring MURI Meeting; Seattle, WA Sharpness (all stations)

4 May :50 AMSpring MURI Meeting; Seattle, WA Sharpness (water only)

4 May :50 AMSpring MURI Meeting; Seattle, WA Minimum 2-m Temperature – Same Story (all stations) (water only) (calibration) (sharpness)

4 May :50 AMSpring MURI Meeting; Seattle, WA Continuous Ranked Probability Scores (all stations) (water only) (MAXT2) (MINT2)

4 May :50 AMSpring MURI Meeting; Seattle, WA Next Steps Go operational with Local-BMA for MAXT2 and MINT2. Code almost ready. Some issues remaining with “blank” grid points. Parameter optimization? Work on precip next (PoP & PQPF). Issues with small training samples and precip. What if all zeroes? Probably need to modify the search parameters. Distance to crest? Up-slope / down-slope? Depends on terrain gradient and wind! Wind (2-D vector). Established methods for wind speed and direction, separately. Use gamma and Von Mises mixture distributions, respectively. Need to build an EM-like algorithm or employ CRPS (energy score) minimization for 2-D wind forecasts. Work is being done on the CRPS (energy score) for 2-D variables. [statistics]

QUESTIONS and DISCUSSION

4 May :50 AMSpring MURI Meeting; Seattle, WA

4 May :50 AMSpring MURI Meeting; Seattle, WA

4 May :50 AMSpring MURI Meeting; Seattle, WA FIT MEC MEC Performance with Grid-Based Verification CRPS = continuous ranked probability score [Probabilistic analog of the mean absolute error (MAE) for scoring deterministic forecasts] Comparison of *UWME 48-h 2-m temperature forecasts: Member-specific mean bias correction applied to both [14-day running mean] FIT = Gaussian fit to the raw forecast ensemble MEC = Gaussian fit to the ensemble-mean + the mean error climatology [00 UTC Cycle; October 2002 – March 2004; 361 cases]

4 May :50 AMSpring MURI Meeting; Seattle, WA MEC Local-BMA Forecast Performance BMA After several attempts to implement BMA with local or regional training data, EXCELLENT results were achieved: when the training data is selected from a neighborhood* of grid points with similar land-use type and elevation Example application to 48-h 2-m temperature forecasts uses only 14 training days. Dramatic improvements in CRPS nearly everywhere. *neighbors have same land use type and elevation difference < 200 m within a search radius of 3 grid points (60 km)

4 May :50 AMSpring MURI Meeting; Seattle, WA An Advanced Calibration Method BMA has several advantages over MEC: A time-varying uncertainty forecast. A way to keep multi-modality, if it is warranted. Maximizes information from short (2-4 week) training periods. Allows for different relative skill between members through the BMA weights (multi-model, multi-scheme physics). Bayesian Model Averaging (BMA) Summary Member-specific mean-bias correction parameters Member-specific BMA weights BMA variance (not-member specific here, but can be) [c.f. Raftery et al. 2005, Mon. Wea. Rev.]

4 May :50 AMSpring MURI Meeting; Seattle, WA For quantities such as wind speed and precipitation, distributions are not only non-Gaussian, but not purely continuous – there are point masses at zero. For probabilistic quantitative precipitation forecasts (PQPF): Model P(Y=0) with a logistic regression. Model P(Y>0) with a finite Gamma mixture distribution. Fit Gamma means as a linear regression of the cubed-root of observation on forecast and an indicator function for no precipitation. Fit Gamma variance parameters and BMA weights by the EM algorithm, with some modifications. Extending BMA to Non-Gaussian Variables [c.f. Sloughter et al. 200x, manuscript in preparation]