1 Probabilistic Forecast Verification Allen Bradley IIHR Hydroscience & Engineering The University of Iowa RFC Verification Workshop 16 August 2007 Salt.

Slides:



Advertisements
Similar presentations
Climate Prediction Applications Science Workshop
Advertisements

Report of the Q2 Short Range QPF Discussion Group Jon Ahlquist Curtis Marshall John McGinley - lead Dan Petersen D. J. Seo Jean Vieux.
1 Verification Continued… Holly C. Hartmann Department of Hydrology and Water Resources University of Arizona RFC Verification Workshop,
“A LPB demonstration project” Celeste Saulo CIMA and Dept. of Atmos. and Ocean Sciences University of Buenos Aires Argentina Christopher Cunningham Center.
14 May 2001QPF Verification Workshop Verification of Probability Forecasts at Points WMO QPF Verification Workshop Prague, Czech Republic May 2001.
Reduction of Streamflow From Upstream Ice Formation a.k.a. “Ice Bite” Brian Connelly North Central River Forecast Center Chanhassen, Minnesota.
Verification and evaluation of a national probabilistic prediction system Barbara Brown NCAR 23 September 2009.
Hydrologic Outlook for the Pacific Northwest Andy Wood and Dennis P. Lettenmaier Department of Civil and Environmental Engineering for Washington Water.
New Local Climate Outlook Products from the NWS Jay Breidenbach NOAA/National Weather Service.
Alan Hamlet Dennis P. Lettenmaier JISAO Climate Impacts Group and the Department of Civil Engineering University of Washington March, 2001 CIG Water Resources.
Regional Climate Change Water Supply Planning Tools for Central Puget Sound Austin Polebitski and Richard Palmer Department of Civil and Environmental.
New Local Climate Outlook Products from the NWS Andrea Bair NOAA/NWS Western Region Headquarters Climate Services Program Manager.
MOS Performance MOS significantly improves on the skill of model output. National Weather Service verification statistics have shown a narrowing gap between.
Alan F. Hamlet Andy Wood Seethu Babu Marketa McGuire Dennis P. Lettenmaier JISAO Climate Impacts Group and the Department of Civil Engineering University.
Western Water Supply Kevin Werner, Andrew Murray, WR/SSD Jay Breidenbach, WFO Boise Cass Goodman, Steve Shumate, CBRFC Alan Takamoto, Scott Staggs, CNRFC.
Ensemble Post-Processing and it’s Potential Benefits for the Operational Forecaster Michael Erickson and Brian A. Colle School of Marine and Atmospheric.
June 23, 2011 Kevin Werner NWS Colorado Basin River Forecast Center 1 NOAA / CBRFC Water forecasts and data in support of western water management.
Colorado Basin River Forecast Center Water Supply Forecasting Method Michelle Stokes Hydrologist in Charge Colorado Basin River Forecast Center April 28,
Evaluation of Potential Performance Measures for the Advanced Hydrologic Prediction Service Gary A. Wick NOAA Environmental Technology Laboratory On Rotational.
Lecture II-2: Probability Review
PROBABILISTIC NUMERICAL GUIDANCE FOR HIGH IMPACT EVENTS (THORPEX) Goal: Provide highest quality, coordinated, and seamless probabilistic automated guidance.
Water Supply Forecast using the Ensemble Streamflow Prediction Model Kevin Berghoff, Senior Hydrologist Northwest River Forecast Center Portland, OR.
Chapter 12 Inferential Statistics Gay, Mills, and Airasian
Reclamation Mid-Term Operational Modeling Seasonal to Year-Two Colorado River Streamflow Prediction Workshop CBRFC March 21-22, 2011 Katrina Grantz, PhD.
Creating Empirical Models Constructing a Simple Correlation and Regression-based Forecast Model Christopher Oludhe, Department of Meteorology, University.
Great Basin Verification Task 2008 Increased Variability Review of 2008 April through July Period Forecast for 4 Selected Basins Determine what verification.
1 Ensemble Reforecasts Presented By: Yuejian Zhu (NWS/NCEP) Contributors:
Seasonal Forecasts in Ethiopia: Hydropower, Ag-Econ & Flood Modeling NASA GHA 1 st Participatory Research Workshop and Project Meeting Addis Ababa 12 August.
Why Normal Matters AEIC Load Research Workshop Why Normal Matters By Tim Hennessy RLW Analytics, Inc. April 12, 2005.
ESET ALEMU WEST Consultants, Inc. Bellevue, Washington.
1 James Brown An introduction to verifying probability forecasts RFC Verification Workshop.
Event-based Verification and Evaluation of NWS Gridded Products: The EVENT Tool Missy Petty Forecast Impact and Quality Assessment Section NOAA/ESRL/GSD.
Assessing the impacts of climate change on Atbara flows using bias-corrected GCM scenarios SIGMED and MEDFRIEND International Scientific Workshop Relations.
National Weather Service Application of CFS Forecasts in NWS Hydrologic Ensemble Prediction John Schaake Office of Hydrologic Development NOAA National.
Overview of the Colorado Basin River Forecast Center Lisa Holts.
Probabilistic Forecasting. pdfs and Histograms Probability density functions (pdfs) are unobservable. They can only be estimated. They tell us the density,
ENSEMBLES RT4/RT5 Joint Meeting Paris, February 2005 Overview of the WP5.3 Activities Partners: ECMWF, METO/HC, MeteoSchweiz, KNMI, IfM, CNRM, UREAD/CGAM,
Educational Research Chapter 13 Inferential Statistics Gay, Mills, and Airasian 10 th Edition.
. Outline  Evaluation of different model-error schemes in the WRF mesoscale ensemble: stochastic, multi-physics and combinations thereof  Where is.
Sub-regional Workshop on Census Data Evaluation, Phnom Penh, Cambodia, November 2011 Evaluation of Age and Sex Distribution United Nations Statistics.
Flash Flood A rapid and extreme flow of high water into a normally dry area, or a rapid water level rise in a stream or creek above a predetermined flood.
CBRFC Stakeholder Forum February 24, 2014 Ashley Nielson Kevin Werner NWS Colorado Basin River Forecast Center 1 CBRFC Forecast Verification.
Logistical Verification Forecast Services in IHFS Mary Mullusky RFC Verification Workshop, August
RFC Climate Requirements 2 nd NOAA Climate NWS Dialogue Meeting January 4, 2006 Kevin Werner.
Distributed Data Analysis & Dissemination System (D-DADS ) Special Interest Group on Data Integration June 2000.
Probabilistic Forecasts of Extreme Precipitation Events for the U.S. Hazards Assessment Kenneth Pelman 32 nd Climate Diagnostics Workshop Tallahassee,
Probabilistic seasonal water supply forecasting in an operational environment: the USDA-NRCS Perspective Tom Pagano
Sources of Skill and Error in Long Range Columbia River Streamflow Forecasts: A Comparison of the Role of Hydrologic State Variables and Winter Climate.
Reforecast Use In Hydrology Hank Herr, Mark Fresch (OHD) James Brown (Hydrologic Solutions Ltd, UK) December 3, 2013.
Alan F. Hamlet Andy Wood Dennis P. Lettenmaier JISAO Center for Science in the Earth System Climate Impacts Group and the Department.
Science plan S2S sub-project on verification. Objectives Recommend verification metrics and datasets for assessing forecast quality of S2S forecasts Provide.
Proposed THORPEX/HEPEX Hydrologic Ensemble Project (THEPS) Presentation for 3 rd THORPEX Science Symposium September 14-18, 2009 Prepared by John Schaake,
Implementing Probabilistic Climate Outlooks within a Seasonal Hydrologic Forecast System Andy Wood and Dennis P. Lettenmaier Department of Civil and Environmental.
DOWNSCALING GLOBAL MEDIUM RANGE METEOROLOGICAL PREDICTIONS FOR FLOOD PREDICTION Nathalie Voisin, Andy W. Wood, Dennis P. Lettenmaier University of Washington,
VERIFICATION OF A DOWNSCALING SEQUENCE APPLIED TO MEDIUM RANGE METEOROLOGICAL PREDICTIONS FOR GLOBAL FLOOD PREDICTION Nathalie Voisin, Andy W. Wood and.
Probabilistic Forecasts - Baseline Products for the Advanced Hydrologic Prediction Services (AHPS) Dave Reed HIC – LMRFC 2006 NWA Annual Meeting October.
EVALUATION OF A GLOBAL PREDICTION SYSTEM: THE MISSISSIPPI RIVER BASIN AS A TEST CASE Nathalie Voisin, Andy W. Wood and Dennis P. Lettenmaier Civil and.
Colorado Basin River Forecast Center Greg Smith Senior Hydrologist National Weather Service Colorado Basin River Forecast Center January 25, 2011 Navajo.
Findings and Recommendations from the Hydrologic Verification System Requirements Team Peter Gabrielsen, Julie Demargne, Mary Mullusky, Kevin Werner, Bill.
Ensemble Forecasts Andy Wood CBRFC. Forecast Uncertainties Meteorological Inputs: Meteorological Inputs: Precipitation & temperature Precipitation & temperature.
RFC Verification Workshop What to Expect Julie Demargne and Mary Mullusky RFC Verification Workshop, August
National Oceanic and Atmospheric Administration’s National Weather Service Colorado Basin River Forecast Center Salt Lake City, Utah 11 The Hydrologic.
HIC Meeting, 02/25/2010 NWS Hydrologic Forecast Verification Team: Status and Discussion Julie Demargne OHD/HSMB Hydrologic Ensemble Prediction (HEP) group.
Verifying and interpreting ensemble products
Eric Jones Senior Hydrologist Lower Mississippi River Forecast Center
Nathalie Voisin, Andy W. Wood and Dennis P. Lettenmaier
Hydrologic ensemble prediction - applications to streamflow and drought Dennis P. Lettenmaier Department of Civil and Environmental Engineering And University.
Long-Lead Streamflow Forecast for the Columbia River Basin for
The Importance of Reforecasts at CPC
Ryan Kang, Wee Leng Tan, Thea Turkington, Raizan Rahmat
Presentation transcript:

1 Probabilistic Forecast Verification Allen Bradley IIHR Hydroscience & Engineering The University of Iowa RFC Verification Workshop 16 August 2007 Salt Lake City

2 Advanced Hydrologic Prediction Service Ensemble streamflow forecasts

3 Advanced Hydrologic Prediction Service Ensemble streamflow forecasts Multiple forecast locations

4 Advanced Hydrologic Prediction Service Ensemble streamflow forecasts Multiple forecast locations Throughout the United States

5 Forecast Location Forecast Date Forecast Variable How good are the ensemble forecasts produced by AHPS? AHPS Verification

6 Outline Illustrate a consistent diagnostic framework for verification of AHPS ensemble forecasts Describe a prototype interactive web- based system for implementing this verification framework within an RFC Present a future vision for the role of verification archives in AHPS forecasting 6

7 Forecast Verification Framework

8 Perspective: Forecast Users Evaluate the quality of forecasts at a specific location for a particular forecast variable and date Examine one “element” in the data cube 8

9 Elemental Problem Use a distributions-oriented approach (DO) to evaluate probability forecasts for “events” defined by a flow threshold Forecast quality attributes quantified over a range of flow thresholds 9

10 Ensemble Forecast

11 Ensemble Forecast Probability forecast of a discrete eventy f

12 Ensemble Forecast Probability forecast of a discrete event Probability forecasts of multicategory eventsWet Near Avg Dry f dry f avg f wet 12

13 Ensemble Forecast Generalize by defining event forecasts as a continuous function of threshold f 0.50 y 0.50 y 0.75 f 0.75 y 0.25 f 0.25 Index function by the threshold’s climatological probability 13

14 Ensemble Forecast Verification Compute forecast- observations pairs for specific thresholds y p Evaluate forecast quality for a range of thresholds y p

15 Des Moines River near Stratford Standard Errors Skill Skill depends on the threshold Uncertainty is greater for extremes April 1 st Forecasts 15

16 Distributions-Oriented Measures Skill Score Decomposition: (SS) Skill (RES) Resolution (CB) Conditional Bias (UB) Unconditional Bias Slope Reliability Standardized Mean Error Potential Skill

17 Implications for Verification Increase Probability forecast skill Eliminate with bias- correction Minimum 7-Day Flow SS RES CB UB 17

18 Perspective: NWS RFC Forecaster Assess the overall performance of the forecasting system Diagnose attributes limiting forecast skill (e.g., biases) Examine “slices” and “blocks” of the data cube 18

19 Multidimensional Problem The forecaster needs summary verification measures suitable for comparing forecasts at different locations and/or forecasts issued on different dates Summary measures describe attributes of the skill functions derived from the elemental verification problem 19

20 Summary Verification Measures Ranked Probability Skill Score (RPSS): Weighted-average skill over probability thresholds

21 Summary Verification Measures Skill RPSS shows average skill Center of mass shows asymmetries in the skill function RPSS Center of Mass 21

22 Hypothetical Skill Functions All skill functions have same average skill 22

23 Hypothetical Skill Functions All skill functions have same average skill Second central moment shows shape 23

24 Hypothetical Skill Functions All skill functions have same average skill Second central moment shows shape 24

25 NCRFC Forecasts 7-day minimum flow forecasts for mainstem locations for three rivers Minnesota River (MIN) Des Moines River (DES) Rock River (RCK) 25

26 Forecast Skill Attributes Forecasts made at the 1 st of the month 26

27 Forecast Skill Attributes Average skill is highest for DES The skill function is peaked in the middle 27

28 Summary Measure Decomposition Skill Score Decomposition: Skill ResolutionConditional Bias Unconditional Bias Weighted-average measures of resolution and biases

29 AHPS Minimum 7-Day Flow A single MIN site has large biases for low flows The largest biases for other sites centered on higher flows 29

30 Forecast Bias Attributes Unconditional bias is dominate Simple bias- correction can significantly improve forecasts Simple bias- correction Post-hoc calibration 30

31 Verification Framework Forecast quality for ensemble forecasts (e.g., skill) is a continuous function of the forecast outcome (or its climatological probability) Summary measures can be interpreted as measures of the “geometric shape” of the forecast quality function This interpretation provides a framework for concisely summarizing the attributes of ensemble forecasts 31

32 AHPS Verification System

33 AHPS Verification System Web-based tools for online access, analysis, and comparison of retrospective AHPS forecasts for River Forecast Centers (RFCs)

34 Map-Based Navigation 2 1 3

35 Verification Data Archive 35

36 Verification Data Archive Retrospective forecasts for a 50-year period 36

37 Verification Data Archive Retrospective forecasts for a 50-year period Processed ensemble forecasts & observations 37

38 Verification Data Archive Retrospective forecasts for a 50-year period Processed ensemble forecasts & observations Verification results 38

39 Verification System Concepts Retrospective ensemble traces available in their native format (*.VS files) Processed ensemble forecasts & observations for a suite of variables Uses *.qme files from the calb system Forecast quality measures based on the ensemble forecasts 39

40 Disk Requirements 6 Forecast periods per month (72 per year) All segments have 50 years observed record

41 Advantages Interactive exploration of verification results Provides a diagnostic “report card” for sites within an RFC Instant access to forecasts and quality measures for verification sites Seamless integration with other components of the NWSRFS system 41

42 A Vision for the Future

43 Vision Generation and archival of retrospective forecasts will be a routine component of forecasting systems Verification methods can assess quality Verification results would form the basis for accepting (or rejecting) proposed improvements to the forecasting system Archival information will form the basis for generating improved forecast products

44 Product Generation with Archive Raw ESP forecast Archive verification indicates biases and skill Optimal merging and bias correction Ensemble Forecast Verification Archive Optimized CS

45 Conclusions

46 Conclusions A consistent verification framework provides both users and forecasters with the means evaluating forecast products (exploring the “data cube”) AHPS-VS integrates retrospective forecast generation and forecast verification within the operational setting of an RFC Retrospective forecast archives will become a routine component of a hydrologic forecasting system, enhancing forecast evaluation and product generation

47

48 Des Moines Forecast Skill Skill is higher (lower) downstream (upstream) Skill decline from April to June JCKM5 DESI4 STRI4 Total Bias