Verification of Precipitation Areas Beth Ebert Bureau of Meteorology Research Centre Melbourne, Australia

Outline 1. “Eyeball” verification - use of maps 2. QPF verification using gridpoint match-ups 3. Space-time verification of pooled data 4. Entity-based (rain “blob”) verification 5. Summary

1. “Eyeball” verification - some examples Accumulated rain over eastern Germany and western Poland, 4-8 July 1997

WWRP Sydney 2000 Forecast Demonstration Project

RAINVAL - Operational verification of NWP QPFs

2. QPF verification using (grid)point match-ups HAll verification statistics can be applied to spatial estimates when treated as a matched set of forecasts/observations at a set of individual points! ObservedForecast Method 1: Analyze observations onto a grid

ObservedForecast Method 2: Interpolate model forecast to station locations Q: Which verification approach is better? A: It depends!

Arguments in favor of grid: point observations may not represent rain in local area gridded analysis of observations better represents the grid- scale values that a model predicts spatially uniform sampling  Use to verify gridded forecasts Arguments in favor of station locations: observations are “pure” (not smoothed or interpolated)  Use to verify forecasts at point locations or sets of point locations Note: Verification scores improve with increasing scale!

Preparation of gridded (rain gauge) verification data: Real time vs. non-real time Quality control to eliminate bad data Mapping procedure: –simple gridbox average –objective analysis (Barnes, statistical interpolation, kriging, splines, etc.) Map observations to model grid Model intercomparison - map to common grid Uncertainty in gridbox values

Continuous statistics quantify errors in forecast rain amount

Categorical statistics quantify errors in forecast rain occurrence

Verification of QPFs from NWP models Vary rain threshold from light to heavy Equitable threat scoreBias score

Verification of NWP QPFs over Germany equitable threat w.r.t. chance equitable threat w.r.t. persistence

Verification of nowcasts in Sydney 2000 FDP —— Nowcast Persistence

3. Space-time QPF verification (a) Pool forecasts and observations in SPACE AND TIME  summary statistics

Caution: Results may mask regional and/or seasonal differences annual winter summer Model performance in Australian tropics

(b)Pool forecasts and observations in SPACE but NOT TIME  maps of temporal statistics No data Bias score June 1995-November 1996

(c)Pool forecasts and observations in TIME but NOT SPACE  time series of spatial statistics OBS LAPS 24 h LAPS 36 h LAPS 48 h 1-30 Apr 2001 Australian region

Limitations to QPF verification using (grid)point match-ups : Some seemingly good verification statistics may result from compensating errors –too much rain in one part of the domain offset by too little rain in another part of the domain –interseasonal rainfall variation captured but shorter period variation not captured Conservative forecasts are rewarded Some rain forecasts look quite good except for the location of the system; unfortunately, traditional verification statistics severely penalize these cases

4. Entity-based QPF verification (rain “blobs”) Verify the properties of the forecast rain system against the properties of the observed rain system: location rain area rain intensity (mean, maximum) ObservedForecast

Define a rain entity by a Contiguous Rain Area (CRA), a region bounded by a user-specified isohyet. Some possible choices of CRA thresholds are: 1 mm d -1 :~ all rain in system 5 mm d -1 :“important” rain 20 mm d -1 : rain center Observed Forecast

Determining the location error: Horizontally translate the QPF until the total squared error between the forecast and the analysis (observations) is minimized in the shaded region. The displacement is the vector difference between the original and final locations of the forecast. Arrow shows optimum shift. Observed Forecast

CRA error decomposition The total mean squared error (MSE) can be written as: MSE total = MSE displacement + MSE volume + MSE pattern The difference between the mean square error before and after translation is the contribution to total error due to displacement, MSE displacement = MSE total – MSE shifted The error component due to volume represents the bias in mean intensity, where and are the CRA mean forecast and observed values after the shift. The pattern error accounts for differences in the fine structure of the forecast and observed fields, MSE pattern = MSE shifted - MSE volume

Example: Nowcasts from Sydney 2000 FDP

Example: Australian regional NWP model

Rain areaMean rain intensity North of 25°S South of 25°S

Maximum rain intensity Rain volume

Displacement error

Event forecast classification Two most important aspects of a “useful” QPF: Location of predicted rain must be close to the observed location Predicted maximum rain rate must be “in the ballpark”

Example: Proposed event forecast criteria for 24h NWP QPFs Good location: Forecast rain system must be within 2° lat/lon or one effective radius of the rain system, but not farther than 5° from the observed location Good intensity: Maximum rain rate must be within one category of observed (using rain categories of 1-2, 2-5, 5-10, 10-25, 25-50, , , , >200 mm d -1 ) Event forecast classification Australian 24h QPFs from BoM regional model, July 1995-June 1999 (2066 events)

Error decomposition Australian 24h QPFs from BoM regional model, July 1995-June 1999 (2066 events)

Advantages of entity-based QPF verification: intuitive, quantifies “eyeball” verification addresses location errors allows decomposition of total error into contributions from location, volume, and pattern errors rain event forecasts can be classified as "hits", "misses", etc. does not reward conservative forecasts Disadvantages of entity-based verification: more than one way to do pattern matching (i.e., not 100% objective forecast must resemble observations sufficiently to enable pattern matching

5. Summary Spatial QPF success* can be qualitatively and quantitatively measured in many ways, each of which tells only part of the story *Note: “success” depends on the requirements of the user!! Objective Subjective PointArea (grid)point match-ups Precision Meaning maps entities