1. Hydrologically Relevant Error Metrics for PEHRPP
Faisal Hossain
Tennessee Technological University
George Huffman
SSAI/GSFC T U
Tennessee Tech
UNIVERSITY C E

2. The Relevant PEHRPP Objective
“To characterize errors in various high resolution precipitation products (HRPP) on many spatial and temporal scales, over varying surfaces and climatic regimes.”
Overland – Hydrologic Application such as flood prediction, soil moisture/ET estimation.
‘Hydrologically Relevant Scale’ ideally, it is the scale needed to solve the hydrologic process equations
Implies scale incongruity between HRPP available at ‘meteorological’ scales (>10 km, >hourly) and the needed smaller scale (~1-5 km,<hourly) for characterizing precipitation variability. T U
Tennessee Tech
UNIVERSITY C E

3. Process-based Understanding of Scale Incongruity
Watershed = Non-linear system yavg ≠ f(xavg)
Time
Space
Non-linearity
Thresholding
An Infiltration Approach to surface runoff modeling (physically-based) as follows: T U
Tennessee Tech
UNIVERSITY C E

4. Implications of Scale Incongruity
Essential Features of Precipitation Error that should matter for overland hydrologic application of HRPPs
“As space and time scales become smaller, satellite precipitation estimation error requires more features to describe its variability.
Fine-scale hydrologic assessment/application requires the recognition of this increasing complexity of satellite precipitation error structure.”
Error Variability in space – spatial structure
Error Variability in time – temporal structure
Scalar Difference in magnitude – retrieval structure T U
Tennessee Tech
UNIVERSITY C E

5. The Question: Are currently used error metrics adequate to pursue PEHRPP objective at Hydrologically Relevant Scales?
What are the widely used metrics for evaluation of HRPPs?
CSI, HSS, ETS, FB, RMSE, POD, FAR (developed originally for weather/climate diagnosis by NWP community)
What do these metrics tell the user on the reliability of a HRPP for a particular overland hydrologic application? – Not yet fully explored
Can they be used in an error model for generation of synthetic HRPP for controlled experiments? – Probably not because they have not yet seen explicit use in error models T U
Tennessee Tech
UNIVERSITY C E

6. The Current Error Models (for synthetic generation of HRPP from validation data)
Mostly related to sampling uncertainty (obviously – because 15 years ago we were concerned about the infrequent MW sampling)
Do Two satellite products with similar RMSE, correlation, bias, ETS value mean similar hydrologic predictability overland? – Not a sufficient condition
Large number of combination of hydrologically-sensitive space-time error structures possible for the same coarse error metrics.
Need to decompose coarse error into various dimensions as much as hydrologically justified to provide feedback beween hydrologists and HRPP developers
Recent trend includes/combines retrieval uncertainty with sampling as sampling frequency has increased
These Error Models provide mathematical expression for an aggregate error metric (typically variance/rmse) T U
Tennessee Tech
UNIVERSITY C E

7. A Proposal for Devising Hydrologically Relevant Error Metrics for PEHRPP
ONE
Hydrologically Relevant Error Metrics should be able to answer three key questions –
Q1. How does the error vary in time?
Q2. How does the error vary in space?
Q3. How “off” is the rainfall estimate from the true value over rainy areas?
TWO
Metrics should have ‘Diagnostic’ and ‘Prognostic’ value
Diagnostic – Able to quantify uncertainty on a specific feature/dimension of precipitation
Prognostic – Amenable for use in a mathematical error model for synthetic generation of HRPPs. T U
Tennessee Tech
UNIVERSITY C E

8. BACKGROUND WORK
Because of intermittency in space and time, four possible outcomes of a HRPP at any given time and space
1. Successful Rain Detection/Delineation (HIT)
2. Unsuccessful Rain Detection/Delineation (MISS)
3. Successful No-Rain Detection/Delineation (HIT)
4. Unsuccessful No-Rain Detection/Delineation (MISS)
Rainy/Non-rainy area delineation (error) has a distinct spatial structure
Systematic retrieval error has a non-negligible spatio-temporal structure
Random retrieval error has a spatial structure.
Regime Dependence of error structure on climate, location, season T U
Tennessee Tech
UNIVERSITY C E

9. One Possible set of Hydrologically Relevant Error Metrics (Error Decomposition)
Probability of rain detection (and as a function of rainfall magnitude of satellite or ‘truth’)
Probability of no-rain detection
First and second order moments of the probability distribution during false alarms.
Correlation length for the detection of rain.
Correlation length for detection of no rain
Systematic retrieval error or mean field bias.
Random retrieval error or error variance.
Correlation length for the retrieval error (conditional, when rain >0.0).
(9) lag-one autocorrelation of the mean field bias. T U
Tennessee Tech
UNIVERSITY C E

10. Response to Spatial Scaling (Persiann)
Probability of detection
WSR-88D Rain rate (mm/hr)
0.04 degree
0.08 degree
0.12 degree
0.16 degree
0.24 degree
0.48 degree
1.0 degree 5 10 15
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
0.04o
0.08o
0.12o
0.16o
0.24o
0.48o
1.0o
Correlation
(Unconditional)
0.386
0.383
0.393
0.401
0.418
0.469
0.569
(Conditional)
0.272
0.298
0.319
0.334
0.361
0.437
0.547
Root Mean Squared Error (mm hr-1)
4.708
3.933
3.530
3.230
2.776
1.98
1.25
Frequency Bias (FB)
1.524
1.405
1.423
1.419
1.460
1.548
1.677
False Alarm Ratio (FAR)
0.686
0.634
0.619
0.601
0.5804
0.537
0.506
Equitable Threat Score (ETS)
0.205
0.235
0.245
0.255
0.271
0.302
0.306
Ksat=Minimum Infiltration Capacity T U
Tennessee Tech
UNIVERSITY C E

11. Suggested Action Items for PEHRPP (For Working Group Discussion)
What additional error metrics do we need at hydrologically relevant scales? What is the Minimum set to cover a broad range of users?
How do we determine adequacy?
Thank You – Phil Arkin and Joe Turk for the ‘Error Metric’ Idea T U
Tennessee Tech
UNIVERSITY C E