1. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 1 Evaluation software tools Cristian Lussana (2) and Michael Borsche (1) (1) Deutscher Wetterdienst, Offenbach, Germany (2) Norwegian Meteorological Institute, Oslo, Norway

2. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 2 Motivation Even regional reanalyses are not perfect Uncertainty estimation is essential How far can we push resolution (before we drown in noise?)

3. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 3 Parameters evaluated and verification scores Precipitation – Time aggregation: daily – scores: EDA, PDF-scores, scale decomposition approach: Intensity-Scale skill score Wind speed (10m to 116m) – Time aggregation: Hourly to monthly – Scores: correlation, bias vs height, variance, seasonal cycle, daily cycle Temperature (2m)

4. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 4 Sharing to code Available on GitHub The “Git” in GitHub. Git is an open-source version control system that was started by Linus Trovalds – the same person who created Linux. The “Hub” in GitHub. Git is a command-line tool, but the center around which all things involving Git revolve – effectively, the Hub, is GitHub.com, where developers can store their projects and network with likeminded people. UERRA-EVA: https://github.com/UERRA-EVA

5. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 5 Evaluation against gridded observational datasets A. Choose a gridded observational dataset (Temp, Prec; daily time res) to be used as a reference for the RRA verification; B. Development of EVA_gridobs, a R-library for RRA spatial verification against gridded datasets: i. Exploratory Data Analysis and PDF-based scores. ii. First experiments on the application of a scale-decomposition technique, the Intensity-Scale skill score, to asses the added value of enhanced resolution in RRAs (SMHI-EURO4m MESAN compared against NGCD)

6. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 6 Conclusions. Wavelet based scale- separation MSE skill-score and scale- separation statistics are: - informative on RRA bias, error and skill on different scales - suitable for comparing models with different resolutions - RRA performances for specific intensity events (thresholded binary fields) Scale-Separation verification technique example: aggregation over JJA 2008

7. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 7 UERRA-EVA: Evaluation software tools https://github.com/UERRA-EVA Assess the added value of enhanced resolution in RRA: wavelet-based scale-separation MSE skill score

8. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 8 GitHub repository: EVA_stationobs https://github.com/UERRA-EVA/EVA_stationobs R-package – reading reanalysis and observation data – calculating and plotting statistical measures – published on GitHub

9. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 9

10. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 10 Scores Correlation, bias, RMSE, anomalies, PDF-score, frequency distribution Contingency table based scores for extreme event analysis: – hit rate, false alarm rate, false alarm ratio, HKS, TS, ETS, frequency bias index, HSS, accuracy, odd ratio, EDI, SEDI

11. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 11 Vertical variability

12. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 12 Storm and calm period

13. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 13 Skill scores 12 skill scores based on contingency table Exceed or fall below a threshold Hit rateFalse alarm rate False alarm ratio True skill Threat scoreEquitable threat score Frequency bias Heidke skill score

14. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 14 Hit rate vs false alarm ratio of hourly means at Hannover Skill scores

15. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 15 Summary Publicly available R-package – Read station and mast tower observations – Read different regional reanalyses – Calculate statistical measures to assess uncertainty – Plot analysis results

16. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 16 Any questions?

17. UERRA user workshop, Toulouse, 3./4. Feb 2016Cristian Lussana and Michael Borsche 17 Questions to the users R scripts ok? Git/github ok? Scores (…) of priority? Verification on local, regional, national, or European scale?