1. Verification has been undertaken for the 3 month Summer period (30/05/12 – 06/09/12) using forecasts and observations at all 205 UK civil and defence aerodromes. Various different verification measures have been calculated to compare the models and explore their strengths and weaknesses. CONTINGENCY TABLES AND RELATIVE OPERATING CHARACTERISTIC CURVE RELIABILITY DIAGRAM COST-LOST ANALYSIS The forecast products are verified using observations from the ATDnet (Arrival Time Difference) system; this is an automatic lightning location network that senses lightning flashes over a geographical area. By timing the arrival of unique very low frequency radio waves (‘Sferics’) that are generated by individual lightning strokes the location of every stroke can be calculated. A stroke count was processed to give the number of strokes within a 50km radius of each airport over a 6 hour period. MOGREPS-GLOBALUKV (post processed) Probability of lightning index exceeding 10 from a 12 member ensemble: 0 → lightning is unlikely 10 → deep convectively unstable environment with lightning a possibility Model resolution: 60km Forecast lead times: T+24 and T+36 Model diagnostics: Precipitation, CAPE, convective cloud features (temperature, base height, depth), wet bulb temperature. Risk of lightning within 50km radius of a location: LR1 → lightning expected LR5 → lightning not expected Model resolution: 1.5km Forecast lead times: T+21 and T+33 Model diagnostics: Lifted index, CAPE, lightning index, total precipitation rate. Forecasts are all valid at 06Z and 18Z (±3hours) © Crown copyright Met Office and the Met Office logo are registered trademarks Met Office FitzRoy Road, Exeter, Devon, EX1 3PB United Kingdom Tel: 01392 885680 Fax: 01392 885681 Email: rebecca.stretton@metoffice.gov.uk The forecasting of hazardous weather conditions such as lightning is of huge importance to the Civil Aviation Authority. The accuracy and skill of lightning forecasts is key to the safety of all flights in the UK. This project looks into the verification of two different models’ lightning products to assess how well they forecast this rare event during the 2012 summer. ForecastsObservations Verification Conclusions and Future Work There were almost 60,000 recorded strikes across the UK within a 13 hour period (05Z to 18Z). Both models indicated lightning was expected with a focus on the East coast of England. The ATDnet observations for 1800Z indicate two strong bands of lightning in that region. Hits 1485 Misses 48 False alarms 11791 Correct rejections 25216 Hits 175 Misses 1359 False alarms 931 Correct rejections 36076 Hit =Non zero probability Figure 3: MOGREPS-Global ROC curve. Assumption: zero probability of precipitable water, minimal cloud cover, minimal risk of lightning. Figure 4: UKV ROC curve. Hit = LR1 and LR2 UKV shows considerably higher hit rate (0.97), however the global model results have a much smaller false alarm rate (0.025). A ROC curve displays the forecasts’ ability to distinguish between an event and a non event. Both ROC curves have a very similar initial gradient showing similar skill where data is present. The limited number of points on the MOGREPS-Global curve is due to the small number of members in the ensemble (lowest threshold = 1/12). The rare nature of lightning focuses the points on the ROC curve closer to the origin. Case Study 28th June 2012 Figure 6: Value plot of MOGREPS-Global and UKV. Perfect score = 1.0. Figure 7: Forecasts for 2012/06/28 18Z: a) MOGREPS-Global T+36, b) UKV T+33, c) lightning stroke map for 15Z to 21Z on 28/06/12. MOGREPS-Global UKV A reliability diagram displays how well the predicted probabilities correspond to their observed frequencies. Figure 5: Reliability diagram of MOGREPS-Global and UKV. A value plot shows the relative improvement in economic value between climatology and a perfect forecast. The UKV curve fully encompasses the MOGREPS-Global curve, indicating UKV has greater value. UKV shows high skill and value, greater than MOGREPS-G, however the specific risk and observed frequencies of LR 1&2 could be explored further. MOGREPS-G does show some skill which shows it has potential given the resolution of the model and the rare nature of lightning. In the future MOGREPS-UK (resolution of 1.5km) will be verified to allow direct comparison of the two models for the same domain and resolution, to determine if an ensemble forecast would be better. The ensemble could also be expanded to use 24 members by combining two forecast runs to investigate the ROC curves of the two models further. Figure 1: Hyperbola plot - the potential locations of the lightning source from every pair of stations, the intersection of all the hyperbolae is the lightning location. Figure 2: Stroke counts within 50km radius of Heathrow and Gatwick airports. Heathrow Gatwick MOGREPS-Global shows good reliability for available probabilities. Due to the categorical nature of the UKV forecast a classic reliability diagram could not be produced. LR3,4,5 indicate minimal risk of lightning which correlates with the near zero observed frequency and LR2&1 indicate some risk of lightning correlating with observations. a) b) c) Verification of Met Office Lightning Forecasts for Aviation Rebecca Stretton, Piers Buchanan, Stephen Moseley, Clare Bysouth