1. Climate Modeling LaboratoryMEASNC State University An Extended Procedure for Implementing the Relative Operating Characteristic Graphical Method Robert J. Mera March 6, 2008 Marine, Earth and Atmospheric Sciences North Carolina State University

2. Climate Modeling LaboratoryMEASNC State University Outline Objective and Motivation Principles of Ensemble Forecasting Traditional Relative Operating Characteristic (ROC) and Economic Value (EV) analysis Extended Relative Operating Characteristic (EROC) Conclusions Applications and Future Work

3. Climate Modeling LaboratoryMEASNC State University Motivation Climate prediction is becoming increasingly important for different sectors of the economy worldwide Courtesy: El Universo A case of false alarm/miss

4. Climate Modeling LaboratoryMEASNC State University Ensemble forecasting Ensemble Prediction System (EPS) forecasting is a method used to account for uncertainties and errors in the forecasting system (recall Chaos theory) Through the ensemble approach one can generate probabilistic forecasts for assessing a future event such as excessive rains, droughts, etc.

5. Climate Modeling LaboratoryMEASNC State University Contingency Matrix NoYes No No cost ( ) Miss ( ) Yes False Alarm ( ) Hit ( ) Observations EPS Forecast Hit Rate: H= δ/( +δ) False Alarm Rate: F= /( + ) A decision maker becomes a user of weather forecasts if he/she alters his/her actions based on forecast information A cost-loss analysis can be assessed based on a 2x2 matrix in which we evaluate the skill of a probabilistic forecast

6. Climate Modeling LaboratoryMEASNC State University Ensemble numbers In our contingency matrix we compute the hit rate and false alarm for an array of ensemble member groups Example: 15 ensembles used – Hit rate and false alarm calculated for only 1 out of 15, 2/15... n/N We can use this information to analyze the skill of a model

7. Climate Modeling LaboratoryMEASNC State University The Relative Operating Characteristic (ROC) The ROC method is widely used for estimating the skill of ensemble prediction systems (EPS) (Marzban, 2004) The closer a curve is to the upper-left-hand corner, the more skillful the forecast system is A perfect forecast system would have a ROC area (ROCA) of 1 A system with no capability of distinguishing in advance between different climate events has a score of 0.5, i.e. lying on the diagonal defined by (0,0) and (1,1) Using the information in our contingency matrix we can compute ROC Each point on the curve represents a group of ensembles (1/15, 2/15, etc.)

8. Climate Modeling LaboratoryMEASNC State University The ultimate utility of climate forecasts is economic and other benefits associated with their actual use in the daily decision-making process of individuals and different organizations Simplistically, users of climate forecasts either DO or DO NOT take action, but the relative value of the forecasts varies with model performance (i.e. hit rate, false alarm) Utility of Climate Predictions

9. Climate Modeling LaboratoryMEASNC State University Economic value (EV) graph based on the ROC graph The economic value (EV) graphical method provides a measure of the EPS performance in relative economic terms for a specific hypothetical range of end users (C/L,) varying from 0 to 1 (Richardson 2000a,b) EV computes the relative economic value using the hit rate and false alarm rate, where a value of 1 translates to a perfect forecast. For the sake of brevity, we will not discuss the mathematics involved.

10. Climate Modeling LaboratoryMEASNC State University EROC Procedure Based on the ROC skill alone it is not possible to determine if a useful level of skill has been achieved for a specific end user. The EV method is more cumbersome to use and its approach could result in some oversimplification of the actual situation for many users because in reality they may have an infinite range of available mitigation options. Also, EV does not provide certain features of ROC that help to diagnose specific characteristics of EPS. Our goal was to develop an alternative procedure similar to the traditional implementation of the ROC graphical method, but one that also provides evaluation for a specific end user.

11. Climate Modeling LaboratoryMEASNC State University ROC and EV relationship ROC measures skill only for the C/L (μ) with the maximum value (V opt ) This effectively ignores any other user

12. Climate Modeling LaboratoryMEASNC State University EROC Specific users are more interested in the economic value related their own mitigation options EROC allows us to build different base lines for different users μ= 0.25 μ= 0.40 μ= V (opt) Two users

13. Climate Modeling LaboratoryMEASNC State University What if the end-user decides to use a forecast only if the V min is at a certain value? EROC allows for this. Initial: V min =0 Initial: V min =0.1 Notice the shift

14. Climate Modeling LaboratoryMEASNC State University Additional advantage Each curve in the EV plot represents a particular group of ensembles on the ROC plot (i.e. a point on the ROC curve) EROC preserves ROCs ability to diagnose each ensemble groups skill and its relative value for a specific user This translates into a possibility of using a smaller number of ensemble members, say 3 or 4 model runs instead of 15 (i.e. a less expensive forecast)

15. Climate Modeling LaboratoryMEASNC State University Conclusions An extended ROC (EROC) procedure has been developed from the traditional ROC and the EV graphical methods used for evaluating the performance of ensemble climate/weather prediction systems In the proposed EROC approach we recommend construction of user-specific baselines that provide us with an analysis of both skill and value of an EPS forecast that is tailor-made for a specific user EROC allows for a clearer picture of minimum value and an ensemble groups skill for a particular user

16. Climate Modeling LaboratoryMEASNC State University Applications and Future Work Implementation of a routine to calculate EROC and EV plots to the CML R statistical library (currently in its final testing stage) You can view the progress on http://climlab.meas.ncsu.edu/erocroutine. html An ongoing project in partnership with NCAR using WRF-DART simulations output for EROC implementation

17. Climate Modeling LaboratoryMEASNC State University Acknowledgements Professor Fred Semazzi Neil Davis Matt Norman Richard Anyah NCSU CLIMLAB http://climlab.meas.ncsu.edu An Extended Procedure for Implementing the Relative Operating Characteristic Graphical Method Fredrick H. M. Semazzi, and Roberto J. Mera Journal of Applied Meteorology and Climatology, Semptember 2006

18. Climate Modeling LaboratoryMEASNC State University Questions?

19. Climate Modeling LaboratoryMEASNC State University

20. Climate Modeling LaboratoryMEASNC State University Criteria for Issuing a forecast Decision to issue a forecast of an event (E) to occur is probabilistically based on the criteria: Where: (N): size of the ensemble (n): number of the runs in the ensemble for which (E) actually occurs (p): probability given by the ratio (n/N) This is the threshold fraction above which the event (E) is predicted to occur based on the model forecast