61 st IHC, New Orleans, LA Verification of the Monte Carlo Tropical Cyclone Wind Speed Probabilities: A Joint Hurricane Testbed Project Update John A. Knaff and Mark DeMaria NOAA - NESDIS - Office of Research and Applications, Fort Collins, CO Chris Lauer NOAA - NWS – Tropical Prediction Center, Miami, FL

61 st IHC, New Orleans, LA TPC Monte Carlo Wind Probabilities Hurricane Gordon Hurricane Helene

61 st IHC, New Orleans, LA Motivation for Verification Assess how good the current wind probability algorithm performs Assess future improvements to the wind probability algorithm Another way to track the performance of the deterministic forecasts

61 st IHC, New Orleans, LA How good are the Monte Carlo wind probabilities? QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? What is the magnitude of the probability forecast errors? What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an event occurred? How well do the predicted probabilities of an event correspond to their observed frequencies? What is the ability of the forecast to discriminate between events and non-events? STATISTICS THAT ANSWER … Multiplicative Bias (half) Brier Score Brier Skill Score Reliability Diagrams Relative Operating Characteristics

61 st IHC, New Orleans, LA 34-kt Probability Grids Hurricane Gordon Hurricane Helene

61 st IHC, New Orleans, LA 34-kt OFCL Forecast Hurricane Gordon Hurricane Helene

61 st IHC, New Orleans, LA 34-kt Best Track Hurricane Gordon Hurricane Helene

61 st IHC, New Orleans, LA How good are the Monte Carlo wind probabilities? QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? STATISTICS THAT ANSWER … Multiplicative Bias

61 st IHC, New Orleans, LA Results: Multiplicative Biases Atlantic, July 18 – October 4, 2006; 105W – 1W, 1N – 60N Range: minus infinity to infinity. Perfect score: 1

61 st IHC, New Orleans, LA How good are the Monte Carlo wind probabilities? QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? What is the magnitude of the probability forecast errors? What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an event occurred? How well do the predicted probabilities of an event correspond to their observed frequencies? What is the ability of the forecast to discriminate between events and non-events? STATISTICS THAT ANSWER … Multiplicative Bias (half) Brier Score Brier Skill Score Reliability Diagrams Relative Operating Characteristics

61 st IHC, New Orleans, LA Results: Brier Skill Score Range: minus infinity to 1, 0 indicates no skill when compared to the reference forecast. Perfect score: 1. Atlantic, July 18 – October 4, 2006; 105W – 1W, 1N – 60N

61 st IHC, New Orleans, LA How good are the Monte Carlo wind probabilities? QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? What is the magnitude of the probability forecast errors? What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an event occurred? How well do the predicted probabilities of an event correspond to their observed frequencies? What is the ability of the forecast to discriminate between events and non-events? STATISTICS THAT ANSWER … Multiplicative Bias (half) Brier Score Brier Skill Score Reliability Diagrams Relative Operating Characteristics

61 st IHC, New Orleans, LA Reliability Diagrams Observed frequency of events Bins of Forecast Probabilities

61 st IHC, New Orleans, LA Results: Reliability Diagrams Atlantic, June 17 – October 4, 2006; 105W – 1W, 1N – 60N

61 st IHC, New Orleans, LA How good are the Monte Carlo wind probabilities? QUESTIONS How does the average forecast magnitude compare to the average observed magnitude? What is the magnitude of the probability forecast errors? What is the relative skill of the probabilistic forecast over the deterministic forecasts, in terms of predicting whether or not an event occurred? How well do the predicted probabilities of an event correspond to their observed frequencies? What is the ability of the forecast to discriminate between events and non-events? STATISTICS THAT ANSWER … Multiplicative Bias (half) Brier Score Brier Skill Score Reliability Diagrams Relative Operating Characteristics

61 st IHC, New Orleans, LA Results: ROC Diagram 2% 5% 10% No skill skill

61 st IHC, New Orleans, LA Results: ROC Skill Score Range: -1 to 1, 0 indicates no skill. Perfect score: 1.

61 st IHC, New Orleans, LA Summary Based upon a preliminary verification of Atlantic Monte Carlo Wind Probabilities Probabilities have small negative biases Probabilities are more skillful than the deterministic (i.e. OFCL) in determining if an event will happen Predicted probabilities appear well related to observed frequencies The Probabilities are skillful in their ability to discriminate between “events” and “non-events”

61 st IHC, New Orleans, LA Next Complete the verification of the wind probabilities in the Atlantic, East Pacific, Central Pacific, and West Pacific Summarize results and identify potential problems with the Monte Carlo software Deliver the verification code to TPC

61 st IHC, New Orleans, LA Future Plans Begin our new project to use the model forecast spread [i.e., the Goerss Predicted Consensus Error (GPCE)] to better estimate track error distributions used for the Monte Carlo integration Assist, where possible, M. Mainelli (TPC) on the understanding the verification of the wind probabilities at the watch/warning break points

61 st IHC, New Orleans, LA Multiplicative Biases (Multiplicative) bias - Answers the question: How does the average forecast magnitude compare to the average observed magnitude? Range: minus infinity to infinity. Perfect score: 1.

61 st IHC, New Orleans, LA Brier Score (half) Brier score - Answers the question: What is the magnitude of the probability forecast errors? Range: 0 to 1. Perfect score: 0.

61 st IHC, New Orleans, LA Brier Skill Score Brier skill score - Answers the question: What is the relative skill of the probabilistic forecast over that of climatology, in terms of predicting whether or not an event occurred? Range: minus infinity to 1, 0 indicates no skill when compared to the reference forecast. Perfect score: 1.

61 st IHC, New Orleans, LA Statistical Methods: Relative Operating Characteristics Forecasts (contingent on the forecast probability) ObservationWarning (W)No Warning (W ’ )Total Event (E)hmE Nonevent (E ’ )fcE’E’ Totalww’w’ N A series of 2x2 contingency tables, which are conditional on a range of forecast probabilities are constructed. For instance a warning would be issued if the probability exceeded 1,2,3,4,…100 % The results of the contingency tables can be quantified in terms of hit rate (hr) = h/(h+m) and false-alarm rate (far)=f/(f+c) A plot of far vs. hr can be created and a skill score created from the area under the curve.

61 st IHC, New Orleans, LA Statistical Methods: ROC diagram & Skill Score,where A is the area under the curve Mason and Graham (1999) skill No skill ROC Skill Score