1. Slide 1 Evaluation of observation impact and observation error covariance retuning Cristina Lupu, Carla Cardinali, Tony McNally ECMWF, Reading, UK WWOSC 2014, Montréal, Canada

2. Slide 2 Outline 1.Motivation 2.Estimating observation error variances 3.Assimilation experiments with an updated diagonal R 4.Summary WWOSC 2014, Montréal, Canada

3. Slide 3 Motivation Current operational ECMWF system is quite complex: ~ 40 millions observations from 60 instruments are daily assimilated. The assumed R and B play an important role in determining the weight of a given observation in the assimilation system. The estimation of the error covariances remains a significant challenge. In assimilation systems, the observation error covariance R describes errors in the observations as well as the forward model; We assume a diagonal R. Diagnostics tools are used to quantify the impact of all observations in ECMWF system both in analysis and forecasts. WWOSC 2014, Montréal, Canada

4. Slide 4 Ways to estimate R Diagnostics based on output from DA systems: Desroziers method (Desroziers et al., 2005) An estimate of the observation error variances may be obtained a posteriori from the statistical analysis of the observation residuals. Adjoint-based methods: makes forecast sensitivity to data assimilation system input parameters [ y, R, x b, B] possible: Forecast sensitivity to observations (FSO) – is used to monitor the impact of observations to reduce short-range forecast errors. Forecast R-sensitivity (Daescu & Todling, 2010; Daescu & Langland, 2013) may be used to provide guidance to error covariance tuning procedures. The sensitivity of a scalar measure of forecast error is computed with respect to changes to a set of covariance parameters. … WWOSC 2014, Montréal, Canada

5. Slide 5 Initial assimilation experiment Aim: Investigate the benefits of an updated (diagonal) R compared to the operational assimilation of IASI/Metop-A Baseline experiment assimilating only conventional observations and IASI /Metop-A with R diagonal as in ECMWF operations. Setup: 4D-Var, T511 (~ 40 km resolution), 137 vertical levels; Period: 8 June – 30 July 2012 The simplified (in terms of observation usage) experiment intends to provide the backbone process for observation error variances tuning. WWOSC 2014, Montréal, Canada

6. Slide 6 Desroziers diagnostic for σ o for IASI The observation error standard deviations (σ o ) assumed in our system are strongly inflated. WWOSC 2014, Montréal, Canada

7. Slide 7 Adjoint-based methods FSO R-sensitivity (FSR) WWOSC 2014, Montréal, Canada FSO: The impact of observation is beneficial in each analysis cycle and reduces 24- h forecast error over the global domain by an average of 23.06%; IASI and AIREP observations are contributing the most to 24-h forecast impact. FSR: Positive sensitivities: identify those observation types whose error variance deflation (decreasing the σ o ) is of potential benefit to the 24-h forecast;

8. Slide 8 FSO IASI (σ o ) 2 – sensitivity guidance WWOSC 2014, Montréal, Canada IASI channels: Positive sensitivities: Long-wave CO 2 temperature- sounding channels; Negative sensitivities: O 3 band (range 1479-1671 ) Inflation of the assigned observation error σ o is of potential benefit to the forecast An observation error σ o specification according with Desroziers estimates may have a detrimental forecast impact. FSO, ch. 1671, O 3 band

9. Slide 9 Adjusting (σ o ) 2 for selected IASI channels WWOSC 2014, Montréal, Canada Use of adjoint-methods for tuning of observation error involved two steps: Use FSR to identify/select IASI channels where observation error standard deviations (σ o ) should be decreased/increased. For selected channels, use Desroziers estimates of (σ o ) to quantify how much this should be changed.

10. Slide 10 Experiments with an updated R Aim: Investigate the benefits of an updated R compared to the operational assimilation of IASI/Metop-A Baseline: assimilating only conventional observations and IASI /Metop-A with R diagonal as in ECMWF operations. Exp.1: As Baseline, but with updated diagonal R for all IASI channels as derived from Desroziers diagnostic; Exp.2: As Baseline, but with updated diagonal R for 33 selected IASI channels (temperature-sounding channels 173- 254 and WV channels 2889-5480). WWOSC 2014, Montréal, Canada

11. Slide 11 IASI AIREP-T Impact on FG-departures WWOSC 2014, Montréal, Canada Normalised by Baseline, 95% confidence interval WIND-U,V Exp.1 Exp.2

12. Slide 12 Forecast scores: geopotential WWOSC 2014, Montréal, Canada Normalised change in RMS geopotential forecast error at 500 hPa Verified against operational analysis; 95% confidence error bars 54 days summer 2012. Normalized difference Exp.1 - Baseline Exp.2 - Baseline Better than Baseline Worse than Baseline Forecast day

13. Slide 13 Total dry energy error norm WWOSC 2014, Montréal, Canada The energy norm evaluates the entire model volume of the atmosphere and calculates a combined error from four meteorological variables. N. Hem. S. Hem. Using the Desroziers diagnosed σ o for all IASI channels results in a degradation of analysis and subsequent forecasts.

14. Slide 14 Analysis sensitivity to observations WWOSC 2014, Montréal, Canada Analysis sensitivityBaselineExp.2 Global Observation influence9.3%10.3% Background influence90.7%89.7%

15. Slide 15 Jo – statistics per channel: IASI WWOSC 2014, Montréal, Canada Current obs. errors (Baseline) Based on effective departure: d eff = (y – H(x)) T R -½ New obs. errors (Exp.2)

16. Slide 16 Forecast R- and B-sensitivities WWOSC 2014, Montréal, Canada Positive R-sensitivities for all observation types : decreasing σ o for all obs. The B-sensitivity provides guidance on weighting in the assimilation system between the background state and the whole observing system: background error covariance inflation. An optimal weighting between B and R information may be explained through a single covariance weight coefficient.

17. Slide 17 Summary Results of a study aimed at tuning observation errors variances for IASI/Metop-A based on two methods: a posteriori diagnosis and adjoint-based R-sensitivity. Using the Desroziers diagnosed σ o for all IASI channels results in a degradation in analysis and subsequent forecasts. Forecast R-sensitivity: found to be promising for providing guidance on IASI channel selection, but does not provide the amount of how much the observation-error variances should be changed. Beneficial forecast impact of geopotential, wind, temperature over the operational R. Forecast R- and B-sensitivities can provide guidance toward the real covariance matrices. The method may show if background information is being over (or under) weighted. In this case it appears the EDA based background errors are overweighting the background. WWOSC 2014, Montréal, Canada

18. Slide 18 Open issues Using Forecast sensitivity to R (FSR) to tune R in the current operational ECMWF system is a challenge: very large number of assimilated observations what modifications to R do we need and why? In the ECMWF system, an ensemble of data assimilations is used to specify background errors. The assumed R is used in the ensemble to perturb observations. Need to investigate the impact of the new R on background error estimate. WWOSC 2014, Montréal, Canada

19. Slide 19 Thank you! Questions? WWOSC 2014, Montréal, Canada