Towards an object-oriented assessment of high resolution precipitation forecasts Janice L. Bytheway CIRA Council and Fellows Meeting May 6, 2015

Introduction NWP models are continuously undergoing improvement Model verification studies determine the skill of model performance Future improvements to NWP relies on knowing not only how well the model performs, but what processes are the cause of a successful or failed forecast Introduce an additional step to verification: Assessment – validation of the model with the intent to determine which variables or model processes are likely related to the model’s performance. CIRA Council and Fellows Meeting, 6 May 20152

Goal Use observations (Stage IV MPE) to verify HRRR model precipitation forecasts with assimilated reflectivities Use object-oriented validation and track features through time to evaluate model performance through forecast period Relate validation to other observations or model variables to determine why model does/does not perform well. CIRA Council and Fellows Meeting, 6 May 20153

Based on the MODE method described in Davis et al. [2006; 2009] Identify precipitating features of interest in model and observations in Central US during 2013 warm season (May-Aug) Apply 15km smoothing to rain field and identify areas where hourly accumulation exceeds a selected threshold (1 mm/hr) Feature is present in observations one hour prior to forecast initialization Maximum observed hourly rainfall 1 hour prior to forecast initialization exceeds 10 mm/hr (obs only) Area within a selected isohyet exceeds 5000 km 2 (obs only) Track features through 15 hours of forecast Validate those observed for at least 70% of the forecast run (12+hours) Find forecast/observed feature pairs at forecast hour 1. Create a database of precipitating features and associated properties. CIRA Council and Fellows Meeting, 6 May Object Oriented Validation

CIRA Council and Fellows Meeting, 6 May HRRR Stage IV

Based on the MODE method described in Davis et al. [2006; 2009] Identify precipitating features of interest in model and observations in Central US during 2013 warm season (May-Aug) Apply 15km smoothing to rain field and identify areas where hourly accumulation exceeds a selected threshold (1 mm/hr) Feature is present in observations one hour prior to forecast initialization Maximum observed hourly rainfall 1 hour prior to forecast initialization exceeds 10 mm/hr (obs only) Area within a selected isohyet exceeds 5000 km 2 (obs only) Track features through 15 hours of forecast Validate those observed for at least 70% of the forecast run (12+hours) Find forecast/observed feature pairs at forecast hour 1. Create a database of precipitating features and associated properties. CIRA Council and Fellows Meeting, 6 May Object Oriented Validation

CIRA Council and Fellows Meeting, 6 May HRRR Stage IV

Based on the MODE method described in Davis et al. [2006; 2009] Identify precipitating features of interest in model and observations in Central US during 2013 warm season (May-Aug) Apply 15km smoothing to rain field and identify areas where hourly accumulation exceeds a selected threshold (1 mm/hr) Feature is present in observations one hour prior to forecast initialization Maximum observed hourly rainfall 1 hour prior to forecast initialization exceeds 10 mm/hr (obs only) Area within a selected isohyet exceeds 5000 km 2 (obs only) Track features through 15 hours of forecast Validate those observed for at least 70% of the forecast run (12+hours) Find forecast/observed feature pairs at forecast hour 1. Create a database of precipitating features and associated properties. CIRA Council and Fellows Meeting, 6 May Object Oriented Validation

CIRA Council and Fellows Meeting, 6 May 2015 Match model feature to observed Do any model features overlap the radar feature? yes no yes Match found Select feature with maximum overlap Do any model objects have centroids within effective radius of observed centroid? no More than one? yes No match exists More than one? no Select feature with most similar total rainfall no Match found 9

Based on the MODE method described in Davis et al. [2006; 2009] Identify precipitating features of interest in model and observations in Central US during 2013 warm season (May-Aug) Apply 15km smoothing to rain field and identify areas where hourly accumulation exceeds a selected threshold (1 mm/hr) Feature is present in observations one hour prior to forecast initialization Maximum observed hourly rainfall 1 hour prior to forecast initialization exceeds 10 mm/hr (obs only) Area within a selected isohyet exceeds 5000 km 2 (obs only) Track features through 15 hours of forecast Validate those observed for at least 70% of the forecast run (12+hours) Find forecast/observed feature pairs at forecast hour 1. Create a database of precipitating features and associated properties. CIRA Council and Fellows Meeting, 6 May Object Oriented Validation

Assigned feature numbers Coordinates of feature center of mass Feature size (area) Feature mean, maximum, and total hourly rainfall PDFs and CDFs of rain rate Use these statistics along with feature maps and masks to calculate Location offset Biases “Standard” validation statistics (FAR, POD, RMSE) CIRA Council and Fellows Meeting, 6 May Stored Attributes

CIRA Council and Fellows Meeting, 6 May Location Offset W E N S

Model Spin-up and Lag Correlations Model takes a few hours to spin up to optimum validation results Model concentrates rainfall into area similar to what was observed at assimilation time, leading to low biases in areal extent. Model appears to over- concentrate assimilated latent heating, resulting in high biases in mean hourly rainfall and maximum hourly intensity. Result is good representation of total system rainfall 0 hour lag 1 hour lag 2 hour lag

Composite Rainfall Feature Forecast Hour 1Forecast Hour 3 Area Bias-67%-22% Mean Bias+61%+25% Max Bias+303%+125% Total Bias+8%+2%

Probability of Precipitation Given 1mm/hr observed CIRA Council and Fellows Meeting, 6 May

TPW and Cloud thickness in near- storm environment CIRA Council and Fellows Meeting, 6 May

Features based validation methods allow evaluation of model performance of specific precipitating objects through time. Even with assimilated radar, 1-2 hours of spin-up before most accurate QPF. HRRR placement relatively good, but eastward propagation may be too slow. HRRR tendency to concentrate convective rainfall in intense cores HRRR appears to require large amounts of moisture to produce moderate rainfall/deep convection. CIRA Council and Fellows Meeting, 6 May Conclusions

Future Potential 3D model output will allow for additional evaluation of performance 2014/2015 HRRR data will allow for comparison with previous years to monitor improvements Use of field projects (IFloodS, IPHEX) for point validation and reference observations of cloud properties Further exploration of results using satellite data Rapidly updated, high resolution geostationary 3D reflectivity profiles from GPM CIRA Council and Fellows Meeting, 6 May

CIRA Council and Fellows Meeting, 6 May

Examining the tails of the PDFs CIRA Council and Fellows Meeting, 6 May