1. DA/SAT Training Course, March 2006 Variational Quality Control Erik Andersson Room: 302 Extension: 2627 erik.andersson@ecmwf.int

2. DA/SAT Training Course, March 2006 Introduction VarQC formalism Rejection limits and tuning Minimisation aspects Examples Summary Plan of Lecture

3. DA/SAT Training Course, March 2006 Introduction (1) Assuming Gaussian statistics, the maximum likelihood solution to the linear estimation problem results in observation analysis weights (w) that are independent of the observed value. Outliers will be given the same weight as good data, potentially corrupting the analysis

4. DA/SAT Training Course, March 2006 Introduction (2) Real histogram distributions of departures (y-Hx b ) can show significant deviations from the pure Gaussian form. They often reveal a more frequent occurrence of large departures than expected from the corresponding Gaussian (normal) distribution with the same mean and standard deviation - showing as wide “Tails”. Observed departure from the background “Tail” QC-rejection? Actual distribution Gaussian K Airep temperature observations

5. DA/SAT Training Course, March 2006 The Normal Jo (1) The normal observation cost function Jo(x) has a quadratic form, which is consistent with the assumption that the errors are Gaussian in nature. If observations errors are uncorrelated then this simplifies to: y: array of observations of length=N x: represents the model/analysis variables H: observation operators R: observation error covariance matrix σ o : observation error standard deviation Normalized departure

6. DA/SAT Training Course, March 2006 The Normal Jo (2) The general expression for the observation cost function is based on the probability density function (the pdf) of the observation error distribution (see Lorenc 1986): p is the probability density function of observation error arbitrary constant, chosen such that Jo=0 when y=Hx

7. DA/SAT Training Course, March 2006 The Normal Jo (3) When for p we insert the normal (Gaussian) distribution (N): we obtain the usual expression In VarQC a non-Gaussian pdf will be used, resulting in a non-quadratic expression for Jo.

8. DA/SAT Training Course, March 2006 VarQC formulation (1) In an attempt to better describe the tails of the observed distributions, Ingleby and Lorenc (1993) suggested a modified pdf, written as a sum of two distinct distributions: A is the prior probability of gross error Normal distribution (pdf), as appropriate for ‘good’ data pdf for data affected by gross errors

9. DA/SAT Training Course, March 2006 VarQC formulation (2) Thus, a pdf for the data affected by gross errors (p G ) needs to be specified. Several different forms could be considered. In the ECMWF implementation (Andersson and Järvinen 1999, QJRMS) a flat distribution was chosen. The reasons for this choice and its consequences will become clear in the following D is the width of the distribution D is here written as a multiple d of the observation error, either side of zero.

10. DA/SAT Training Course, March 2006 VarQC formulation (3) Inserting p QC for p in the expression Jo=-ln p + const, we obtain: We can see how the presence of γ modifies the normal cost function and its gradient

11. DA/SAT Training Course, March 2006 Probability of gross error The term modifying the gradient (on the previous slide) can be shown to be equal to: the a-posteriori probability of gross error P, given x and assuming that Hx is correct (see Ingleby and Lorenc 1993) Furthermore, we can define a VarQC weight W: It is the factor by which the gradient’s magnitude is reduced. Data which are found likely to be incorrect (P≈1) are given reduced weight in the analysis. Data which are found likely to be correct (P ≈ 0) are given the weight they would have had using purely Gaussian observation error pdf.

12. DA/SAT Training Course, March 2006 Illustrations flat p QC (left), wide Gaussian p QC (right) Gradient QC Weight

13. DA/SAT Training Course, March 2006 Application In the case of many observations, all with uncorrelated errors, Jo QC is computed as a sum (over the observations i) of independent cost function contributions: The global set of observational data includes a variety of observed quantities, as used by the variational scheme through their respective observation operators. All are quality controlled together, as part of the main 4D-Var estimation. The application of VarQC is always in terms of the observed quantity.

14. DA/SAT Training Course, March 2006 Minimisation VarQC requires a good ‘preliminary analysis’. Otherwise incorrect QC-decisions will occur. In operations VarQC is therefore switched on after 40 iterations. (Now 40)

15. DA/SAT Training Course, March 2006 Multiple minima The probability of gross error depends on the size of the departure. When the probability for rejection is close to 50/50 this will be reflected in the cost function as two distinct minima of roughly equal magnitude.

16. DA/SAT Training Course, March 2006 Rejection limits VarQC does not require the specification of threshold values at which rejections occur - so called rejection limits. Rejections occur gradually. If, for example, we classify as rejected those data that have P>0.75, then we can obtain an analytical relationship for the effective rejection limit as a function of the two VarQC input parameters A and d (or γ) only: The first implementation of VarQC was thereby tuned to roughly reproduce the rejections of the old scheme (OIQC).

17. DA/SAT Training Course, March 2006 Tuning the rejection limit The histogram on the left has been transformed (right) such that the Gaussian part appears as a pair of straight lines forming a ‘V’ at zero. The slope of the lines gives the Std deviation of the Gaussian. The rejection limit can be chosen to be where the actual distribution is some distance away from the ‘V’ - around 6 to 7 K in this case, would be appropriate.

18. DA/SAT Training Course, March 2006 Tuning example BgQC too tough BgQC and VarQC correctly tuned The shading reflects the value of P, the probability of gross error

19. DA/SAT Training Course, March 2006 Example (1) For given values of the VarQC input parameters (A and d), the QC result (i.e. P), is a function of the normalized departure (y-Hx)/σ o, only.

20. DA/SAT Training Course, March 2006 Example (2) VarQC checks all data and all data types simultaneously. In this Australian example the presence of aircraft data has led to the rejection of a PILOT wind.

21. DA/SAT Training Course, March 2006 Example (3) - a difficult one Observations of intense and small-scale features may be rejected although the measurements are correct. The problem occurs when the resolution of the analysis system (as determined by the B-matrix) is insufficient.

22. DA/SAT Training Course, March 2006 Summary VarQC provides a satisfactory and very efficient quality control mechanism - consistent with 3D/4D-Var. The implementation can be very straight forward. VarQC does not replace the pre-analysis checks - the checks against the background for example. All observational data from all data types are quality control simultaneously, as part of the general 3D/4D-Var minimisation. The setting of VarQC parameters needs regular revision. A good description of background errors is essential for effective, flow-dependent QC