Hydrologically Relevant Error Metrics for PEHRPP Faisal Hossain Tennessee Technological University George Huffman SSAI/GSFC T U Tennessee Tech UNIVERSITY C E

The Relevant PEHRPP Objective “To characterize errors in various high resolution precipitation products (HRPP) on many spatial and temporal scales, over varying surfaces and climatic regimes.” Overland – Hydrologic Application such as flood prediction, soil moisture/ET estimation. ‘Hydrologically Relevant Scale’ ideally, it is the scale needed to solve the hydrologic process equations Implies scale incongruity between HRPP available at ‘meteorological’ scales (>10 km, >hourly) and the needed smaller scale (~1-5 km,<hourly) for characterizing precipitation variability. T U Tennessee Tech UNIVERSITY C E

Process-based Understanding of Scale Incongruity Watershed = Non-linear system yavg ≠ f(xavg) Time Space Non-linearity Thresholding An Infiltration Approach to surface runoff modeling (physically-based) as follows: T U Tennessee Tech UNIVERSITY C E

Implications of Scale Incongruity Essential Features of Precipitation Error that should matter for overland hydrologic application of HRPPs “As space and time scales become smaller, satellite precipitation estimation error requires more features to describe its variability. Fine-scale hydrologic assessment/application requires the recognition of this increasing complexity of satellite precipitation error structure.” Error Variability in space – spatial structure Error Variability in time – temporal structure Scalar Difference in magnitude – retrieval structure T U Tennessee Tech UNIVERSITY C E

The Question: Are currently used error metrics adequate to pursue PEHRPP objective at Hydrologically Relevant Scales? What are the widely used metrics for evaluation of HRPPs? CSI, HSS, ETS, FB, RMSE, POD, FAR (developed originally for weather/climate diagnosis by NWP community) What do these metrics tell the user on the reliability of a HRPP for a particular overland hydrologic application? – Not yet fully explored Can they be used in an error model for generation of synthetic HRPP for controlled experiments? – Probably not because they have not yet seen explicit use in error models T U Tennessee Tech UNIVERSITY C E

The Current Error Models (for synthetic generation of HRPP from validation data) Mostly related to sampling uncertainty (obviously – because 15 years ago we were concerned about the infrequent MW sampling) Do Two satellite products with similar RMSE, correlation, bias, ETS value mean similar hydrologic predictability overland? – Not a sufficient condition Large number of combination of hydrologically-sensitive space-time error structures possible for the same coarse error metrics. Need to decompose coarse error into various dimensions as much as hydrologically justified to provide feedback beween hydrologists and HRPP developers Recent trend includes/combines retrieval uncertainty with sampling as sampling frequency has increased These Error Models provide mathematical expression for an aggregate error metric (typically variance/rmse) T U Tennessee Tech UNIVERSITY C E

A Proposal for Devising Hydrologically Relevant Error Metrics for PEHRPP ONE Hydrologically Relevant Error Metrics should be able to answer three key questions – Q1. How does the error vary in time? Q2. How does the error vary in space? Q3. How “off” is the rainfall estimate from the true value over rainy areas? TWO Metrics should have ‘Diagnostic’ and ‘Prognostic’ value Diagnostic – Able to quantify uncertainty on a specific feature/dimension of precipitation Prognostic – Amenable for use in a mathematical error model for synthetic generation of HRPPs. T U Tennessee Tech UNIVERSITY C E

BACKGROUND WORK Because of intermittency in space and time, four possible outcomes of a HRPP at any given time and space 1. Successful Rain Detection/Delineation (HIT) 2. Unsuccessful Rain Detection/Delineation (MISS) 3. Successful No-Rain Detection/Delineation (HIT) 4. Unsuccessful No-Rain Detection/Delineation (MISS) Rainy/Non-rainy area delineation (error) has a distinct spatial structure Systematic retrieval error has a non-negligible spatio-temporal structure Random retrieval error has a spatial structure. Regime Dependence of error structure on climate, location, season T U Tennessee Tech UNIVERSITY C E

One Possible set of Hydrologically Relevant Error Metrics (Error Decomposition) Probability of rain detection (and as a function of rainfall magnitude of satellite or ‘truth’) Probability of no-rain detection First and second order moments of the probability distribution during false alarms. Correlation length for the detection of rain. Correlation length for detection of no rain Systematic retrieval error or mean field bias. Random retrieval error or error variance. Correlation length for the retrieval error (conditional, when rain >0.0). (9) lag-one autocorrelation of the mean field bias. T U Tennessee Tech UNIVERSITY C E

Response to Spatial Scaling (Persiann) Probability of detection WSR-88D Rain rate (mm/hr) 0.04 degree 0.08 degree 0.12 degree 0.16 degree 0.24 degree 0.48 degree 1.0 degree 5 10 15 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 0.04o 0.08o 0.12o 0.16o 0.24o 0.48o 1.0o Correlation (Unconditional) 0.386 0.383 0.393 0.401 0.418 0.469 0.569 (Conditional) 0.272 0.298 0.319 0.334 0.361 0.437 0.547 Root Mean Squared Error (mm hr-1) 4.708 3.933 3.530 3.230 2.776 1.98 1.25 Frequency Bias (FB) 1.524 1.405 1.423 1.419 1.460 1.548 1.677 False Alarm Ratio (FAR) 0.686 0.634 0.619 0.601 0.5804 0.537 0.506 Equitable Threat Score (ETS) 0.205 0.235 0.245 0.255 0.271 0.302 0.306 Ksat=Minimum Infiltration Capacity T U Tennessee Tech UNIVERSITY C E

Suggested Action Items for PEHRPP (For Working Group Discussion) What additional error metrics do we need at hydrologically relevant scales? What is the Minimum set to cover a broad range of users? How do we determine adequacy? Thank You – Phil Arkin and Joe Turk for the ‘Error Metric’ Idea T U Tennessee Tech UNIVERSITY C E