1. Craig H. Bishop Elizabeth A Satterfield Kevin T. Shanley, David Kuhl, Tom Rosmond, Justin McLay and Nancy Baker Naval Research Laboratory Monterey CA November 2, 2012 Hidden Error Variances and the optimal combination of static and flow dependent variances 1

2. Introduction: Definitions Error Variance: Mean of a large number of squared forecast errors. Flow Dependent Error Variance: Mean of a large number of squared forecast errors given a particular flow. (In order to obtain a large number of errors the “flow” or “condition” must repeat itself). Hidden Error Variance: A flow dependent error variance that is formally unobservable because the particular flow does not repeat itself. “A conundrum of predictability research is that while the prediction of flow dependent error distributions is one of its main foci, chaos hides flow dependent forecast error distributions from empirical observation.” Bishop and Satterfield (2012a,b, MWR, in press), Satterfield and Bishop (2012ab, to be submitted)

3. Binned Ensemble variance (mm^2) Spread-skill plot for COAMPS simulations relative to the control (solid line) and Mean (dashed line) 0200400500 600 45 o 0 100 200 300 400 500 600 ET-ctl ET-ens 300100 48-h total precipitation (mm) Binned squared Error (mm)^2 Mean ET-ctl = 119.41 Mean ET-ens = 100.82 Spread-skill plot for COAMPS simulations similar to for 48-h total accumulated precipitation (mm)^2. Significant spread-skill relationships were found for all variables – including precipitation Previous work: spread-skill diagrams Collect innovations (ob – fcst) corresponding to similar ensemble variances into a bin. Compute bin averaged squared innovation. It should increase with ensemble variance. These diagrams do not reveal (a)the climatological range of true error variances, nor (b)the degree of variation of ensemble variance given a true error variance. 3

4. Overview Observations of hidden error variances using replicate systems. Empirical determination of key pdfs. Analytic model of statistical relationships of ensemble variances and true error variances. Use of (innovation, ensemble-variance) pairs to estimate parameters of analytic model. Estimation of optimal weights for Hybrid. Comparison of performance of Hybrid DA with weights from brute force tuning and weights from hidden error variance theory. Other Hybrid results Conclusions 4

5. What is the true flow dependent error variance ? Imagine an unimaginably large number of quasi-identical Earths. (Slartibartfast – Magrathean designer of planets, D. Adams, Hitchhikers …) 5

6. 25000 Lorenz Model Replicates Reveal Hidden Error Variance Using a 10 variable Lorenz ’96 model with additive model error and a 20 member Ensemble Transform Kalman Filter (ETKF) data assimilation scheme, we created 25,000 independent time series of analyses and forecasts, each having the same true state but differing random draws of observation error. True Error Variances were then obtained for each spatio-temporal point by averaging the squared forecast error for this point across the 25,000 replicates. First demonstration of ETKF accurately predicting true flow dependent error variance in non-linear system. Scatter plot of ETKF ensemble variance from a single replicate system as a function of true error variance. The true error variance is estimated from all 25,000 replicate systems. The linear fit to the points on the scatter plot is governed by the equation. 6

7. Controlled accuracy of ensemble variances by degrading ETKF variances A primary objective is to show how pdf of true error variances given an imperfect ensemble variance changes as the accuracy of the ensemble variance changes. To do this, we created degraded ensemble variances by sampling a Gamma distribution with mean equal to the ETKF variance and relative variance determined by an “effective ensemble size” M. (b) M=4 (a) M=8 Examples of assumed likelihood gamma pdfs of ensemble variances with a mean of unity. Panel (a) is for an effective ensemble size of M=8, or equivalently, a relative variance of 2/7. Panel (b) is for an effective ensemble size of 4, or equivalently, a relative variance of 2/3. 7

8. Histograms of true error variance given an imperfect ensemble variance The histograms give an empirical estimate of the pdf of true error variances given a constrained range of sample variances for an 8 member ensemble. The ranges are given on each figure; they correspond to the 2nd and 34th bins, respectively, of 35 bins of true error variance. The solid lines give the fit of an inverse-gamma function to the distribution of true error variances in each bin. Inverse-gamma distribution is a very good fit to empirically derived histogram of true error variances given an ensemble variance for all ensemble variance categories. M=8 8

9. Climatological pdf of true error variances Prior climatological distribution of true error variances. Bars show the probability density histogram of forecast error variances. Solid line shows the fit of the pdf (eq 4) to the data. The thick dashed line marks the mean of both the pdf and the data. M=8 Inverse-gamma distribution gives a reasonable fit to empirically derived prior climatological pdf of true error variances. 9

10. Empirical estimation of pdf of true error variance given ensemble variance from 25000 trials (a) M=8, empirical(b) M=2, empirical Red lines depict empirical estimate of pdf of true error variance (ordinate axis) given fixed values of ensemble variance (abscissa axis). Thin green and blue lines give the mode and mean of the empirical estimates of the mode and mean of these estimates. Panels (a) and (b) show the empirical estimates for random sample ensembles of sizes M=2 and M=8, respectively. The grey shading gives an inverse- gamma pdf fit to the climatological pdf of true error variances. 10

11. An analytic model of hidden error variance Assumption 1: The error of the deterministic forecast is a random draw from a Gaussian distribution, whose true variance  i 2 is a random draw from a prior climatological inverse gamma pdf of error variances.

12. An analytic model of hidden error variance Assumption 2: Ensemble variances are drawn from a likelihood gamma pdf of ensemble variances with mean a(  i 2 -  min 2 )+s 2 min stochastic (b) M=4 (a) M=8

13. An analytic model of hidden error variance Bayes’ Theorem defines the posterior inverse gamma pdf of error variances given an imperfect ensemble variance s i 2 Climatological Prior Distribution Likelihood distribution of s 2 given a particular  2 13

14. (a) M=8, empirical (c) M=8, analytic(d) M=2, analytic (b) M=2, empirical Green lines give mode Blue lines give mean Given an ensemble variance, there are a broad range of possible true error variances. Current DA schemes require a single value. For the minimum error variance estimate, use the posterior mean. For the maximal likelihood estimate, … For QC, … 14

15. Posterior mean error variance is a Hybrid combination of static and ensemble variances Flow dependent ensemble variance Static climatological mean error variance i.As the stochastic variation of ensemble variance about the true variance goes to zero, the weight on the ensemble variance goes to 1. ii.If there is any imperfection in the flow-dependent ensemble variance, the optimal error variance estimate gives weight to the climatological covariance. iii.If there is no variance of the true error variance, the weight on the static variance goes to 1. Purely flow dependent error variance models are sub-optimal Implications for Ensemble DA? Implications for 4DVAR? 15

16. Problem 16

17. Solution: Equations that define hidden parameters from data assimilation output 17

18. Retrieved hidden parameters, var(s 2 ), s 2, a and M are shown in plots (a), (b), (c) and (d), respectively “Light grey bars: M=2, Dark grey bars: M=8 The “given” ensemble sizes in (d) are the random sample ensemble sizes used to degrade the quality of the ETKF ensemble variance. Equations recover hidden parameters “observed” by replicate systems “Observed” values are obtained from 175 DA cycles of the 25,000 “replicate systems” Minimum, mean and maximum of the values retrieved from 21 single system independent time series of with n=2,000,000.

19. Each plot summarizes information from 60 independent retrievals. The values marked as min, mean, max and std are the minimum, mean, maximum and standard-deviation of the values retrieved from 60 completely independent synthetically generated data sets. Recovery of min(sigma^2) is inaccurate when min(sigma^2) is small These tests pertain to synthetic data generated using the analytical model of hidden error variance The “specified” were set equal to values previously retrieved from Lorenz model experiments with a “given” M=8 and differing values of the model error q. Each retrieval is from 2,000,000 (innovation, ensemble-variance) pairs synthetically generated from specified distributions.

20. Variation of weights for mean of posterior distribution of true error variances with model error q and given effective ensemble size M. Black bars give the weights for the de- biased flow-dependent ensemble variance while grey bars give the corresponding weights for the static mean of the climatological error variances. Variation of optimal weights with model error and ensemble size, M Ensemble variance weight in dark grey. Static variance weight in light grey. q gives model error variance parameter M gives an “effective ensemble size” corresponding to the relative variance of a random normal ensemble of size M. The weight on the ensemble variance increases with ensemble size The weight on the static variance increases as model error variance increases 20

21. Use of recoveries in Hybrid DA Flow-dependent ensemble prediction Covariance matrix of unavoidable errors(?) Static covariance 21 As a start, let’s guess that the optimal weights for error variance prediction are “useful” weights for error covariance prediction.

22. Application to Hybrid DA: Lorenz model 1, perturbed observations. A suboptimal M=32 member ensemble is generated using a perturbed observations update. A climatological error covariance matrix (P f climatology ) is formed by collecting forecast errors for 100,000 time steps (using an 100% ensemble based error covariance matrix) P f hybrid is computed at each time step and used in the ETKF DA scheme to obtain an analysis, which is cycled. We compute the “best practice” hybrid and the “standard” hybrid for all alpha values for comparison. “Best Practice” hybrid: The ensemble based P f is corrected by a factor of Hybrid based on weights from theory performs as well as that obtained from brute force tuning of the weights.

23. The eq’s include a kurtosis term which is likely to be sensitive to data QC decisions based on the size of innovations. Fortunately, it may be shown that the weight for the ensemble variances is entirely independent of this term. The weight for the static term can then be obtained by insisting that the average of Hybrid variance be consistent with innovation variance. 23 Possible approaches to concerns in application of theory to Hybrid DA

24. NAVDAS-AR-Hybrid Results Low resolution results Hybrid weights computed for 6 distinct regions using new theory Alpha=0.5 Hybrid based on weights from theory performs as well as that obtained from brute force tuning of the weights.

25. NAVDAS-AR-Hybrid Results High Resolution Results alpha=.5 vs alpha=0 RMS wind error radiosonde verification results. Red means Hybrid outperformed non- Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011. (From Kuhl et al. 2012, in review)

26. NAVDAS-AR-Hybrid Results High Resolution Results alpha=.5 vs alpha=0 RMS wind error self-analysis verification results. Red means Hybrid outperformed non- Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011. (From Kuhl et al. 2012, in review)

27. NAVDAS-AR-Hybrid Results High Resolution Results alpha=.5 vs alpha=0 RMS error global radiosonde verification results. Red means Hybrid outperformed non- Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011. (From Kuhl et al. 2012, in review)

28. NAVDAS-AR-Hybrid Results High Resolution Results alpha=.5 vs alpha=0 Geopotential Height Anomaly Correlation (verification against self-analysis). Red means Hybrid outperformed non-Hybrid. 0000 UTC 1 February 2011 to 0000 UTC 1 April 2011. (From Kuhl et al. 2012, in review)

29. Conclusions 1.A simple theory of the relationships between ensemble variances and true error variances has been developed. 2.This theory provides a new method for estimating from an archive of (innovation, ensemble-variance) pairs a.the prior climatological pdf of true error variances, b.the likelihood pdf of ensemble variances given a true error variance, c.the posterior pdf of true error variances given an ensemble variance, and d.the mean of (c) as a weighted sum of a static and an ensemble variance. 3.The pdfs 2a and 2c are well approximated by inverse-gamma distributions for the Lorenz 96 system. 4.Result (2d) provides a theoretical justification for Hybrid error covariance models that linearly combine static and flow-dependent covariances. 5.The ansatz that the optimal weights (2d) for error variances are useful for error covariances is true for the Lorenz ’96 system with perturbed obs Hybrid DA and a low resolution Hybrid-4DVAR version of the Navy’s operational DA scheme. 6.Enables Hybrid weights to be defined regionally at a fraction of the cost of weights obtained via trial and error. 7.QC and ensemble post-processing applications are also possible. 29