© Crown copyright Met Office Operational OpenRoad verification Presented by Robert Coulson

© Crown copyright Met Office Operational OpenRoad verification Introduction Verification Methodology Temporal Verification Economic skill in forecasts Questions and Answers

© Crown copyright Met Office Operational OpenRoad Verification Site specific verification. > 300 sites are verified. Observations from roadside sensors are used as TRUTH against which the forecasts are verified. A large sample size of matching forecast/observation pairs are collected over individual months and for the whole winter season for each site to ensure validation is reliable OpenRoad is a weather forecasting package that helps road maintenance decision makers to plan, manage and minimise the effects of winter weather. Road travel on a timely basis is essential for economies globally in the modern world. Cold winter climates impact on safety with dangerous travelling conditions causing delays if not acted upon. Therefore, the prediction and prevention of ice formation is imperative to mitigate against these. This presentation shows examples of the verification carried out for OpenRoad classic. The benefits of winter maintenance have been estimated at approximately eight times costs of prevention. In the UK, which has periods of marginal temperatures during the winter months, an estimated £150 million pounds per year (at about £1250 per km) has been spent historically (SIRWEC 2006). The estimated world expenditure on winter maintenance was put at £6 billion (€10 billion) (SIRWEC 2006), and clearly the realised benefit from this is significantly material. Thus there is a strong link for commercial customers to use accurate weather forecasts in order to save money. Hence strict forecast accuracy criteria are agreed in customer contracts.

© Crown copyright Met Office Verification Methodology Categorical verification of road surface frosts 2x2 tableObserved FrostNo Observed Frost No Forecast Frost Miss (c)Correct Rejection (d) Forecast FrostHit (a)False Alarm (b) The critical event is when the road surface temperature is less or equal to zero degrees Celsius due to formation of ice. The observed and forecast road surface temperatures between 12Z UTC and 12Z UTC the next day are collated. A single minimum forecast and observed temperature pair are recorded and stored for this time period (Ob, Fc).

© Crown copyright Met Office Communication to end users 2x2 tableObserved FrostNo Observed Frost No Forecast Frost Miss (c)Correct Rejection (d) Forecast FrostHit (a)False Alarm (b)

© Crown copyright Met Office Temporal Verification Experiments have shown that four times more salt is required to melt snow and ice than to prevent its initial formation on road surfaces. Conversely, if the treating agent is applied to the road surface too soon, traffic and precipitation may remove it. The predicted time of road surface temperature threshold crossing is critical for effective and economical maintenance of transport networks.

© Crown copyright Met Office N by M contingency table The road surface sensors enable the actual state of the road surface to be deduced, and this enables a time-series profile to be verified against the forecast road state. Uncertainty in matching observations to forecasts as the road surfaces are being treated live.

© Crown copyright Met Office Economic skill values for forecasts Action taken ? Event Occurs ? Frost ObservedNo Frost Observed TreatedCost1 incurred Cost2 incurred NOT Treated Loss incurred Null cost Based on the paper by D.S. Richardson, Skill and relative economic value of ECMWF ensemble prediction system, Q.J.R. Meteorol. Soc. (2000),126, pp Defines the relative value V of a forecast system using The maximum value of V is given by the Kuiper’s performance index = Hit Rate – False Alarm Rate If a perfect knowledge of the future weather will save the user an amount S (over the use of purely climatological information) then forecast system with relative value V will save the user 100V% of S Relative economic value The maximum value of V =1 for perfect forecasts and if V>0 then the user will obtain benefit from the forecasts. ME(Climate) = Mean expense by using climate as a forecast. ME(forecast) = Mean expense by using forecasts. ME(perfect) = Mean expense of perfect forecasts. Cost loss ratio = (cost of taking the action)/(potential loss protected by the action) Often the potential loss protected by the action is taken to equal to the total loss that would have occurred. Also known as the Peirce’s Skill Score. It is an equitable skill score, which means that constant forecasts and random forecasts all give a value of zero. It also has no dependence on sample climate. Good forecasts have values with scores closer to 1, but a range of values for the score can be between -1 to +1.

© Crown copyright Met Office Economic skill values for forecasts For marginal nights, - replacing the mean expense from climatology with the expense of salting every marginal night - using a cost to loss ratio = (Thornes 1999) as the benefits of winter maintenance have been estimated to about eight times the cost. Marginal Night analysis The Value Index is based on the one used by Thornes and Stephenson (2001). E(S) = Expense of salting every marginal night. E(A) = Expense of salting with actual forecasts. E(P) = Expense with perfect forecasts. Cost:loss ratio is the cost of taking action (i.e. salting the roads) as a fraction of the loss incurred (i.e. the roads are not salted and accidents and delay occur) taking into account the savings made by not salting. It is assumed 0<cost:loss < 1. ValueIndex = {E(S)-E(A)}/{E(S)-E(P)} ValueIndex(AlwaysProtect) = {[(c+d)-(c/(cost:loss))]/[b+d]} Verification of cost-loss models are evaluated as a consequence.

© Crown copyright Met Office Questions and answers