Sarah Dance DARC/University of Reading Data Assimilation Theory and Practice: Lessons learned from the Atmosphere Carbon Fusion Meeting May 2006 Sarah Dance DARC/University of Reading

Outline Data assimilation for NWP Example: a choice of methods Implementation matters! Summary and conclusions

Some Data Assimilation Applications 1. Retrospective analysis -- Learn more about how the Earth works, by using models to interpret/extend different types of data (e.g., climate studies, atmospheric chemistry) 2. Diagnosis – Test and improve models (e.g., parameter estimation) 3. Forecasting – Use recent data to improve initial conditions for short-term predictions (e.g., numerical weather forecasting) 4. Real-time Control – Use continually changing estimates of system state to determine control actions (e.g. traffic flow) Smoothing: Update states at all times with all meas. Retrospective analysis, diagnosis t Filtering: Update states only with previous measurements Forecasting/ real time control t

Re-analysis of ozone data from MIPAS on Envisat

Initial conditions for NWP Observations can provide some (not all) required information – O(106) obs assimilated every 6 hours Observations can provide some information, but not all. Quality control. Observation model relates state variables to observables e.g. interpolation, satellite radiances Observation errors are usually assume unbiased, conditionally independent, spatially uncorrelated, Gaussian Images from Chilbolton, UFAM, BBC, Met Office, EUMETSAT

Forecast models Global (60km)  Convective-scale (1-4km) Dynamics is very different on these scales Length of DA state vector O(107) variables Want compatible initial conditions that are a solution of the NWP model: avoid unslaved gravity modes spin-up of hydrological cycle

Example: A choice of methods 4D-Var vs EnKF Cf. Lorenc (2003) QJRMS

Measurement model Not all observation operators are linear. Indeed some are not even differentiable e.g. scatterometer winds

4D-Var Incremental method – uses linearization and adjoint of M and H. Converges to nonlinear 4D-Var solution (conditions apply). (Lawless et al. 2004) Minimum of nonlinear 4D-Var may not be unique. Scatterometer ob, identity map model, B=1.0, R=0.1, xb =0.0, y=0.99 However, other obs may remove the ambiguity Incremental 4D-Var is like a quasi-Newton method and hence has similar convergence results. There are other nonlinear optimization methods, such as simulated annealing, that do not require the adjoint and TLM etc, which may be more suited in this case, but alos more computationally expensive.

1D “Scatterometer” winds Not invertible Not differentiable yo = 0.5 d = yo –H(x) S’pose K has a positive sign. Then any ensemble members lying in the region x < -0.5 will be moved away from the observation in the update step. Furthermore, the members which are furthest away from the observation in the first place will have the largest increments away from the observations, leading to filter divergence. EnKF increments = K d K is scalar d < 0 d < 0 d >0 d >0

EnKF with 1D scatterometer x0 ~ N(0.4,1.0) xt =0.5 X observations ensemble Perturbed obs EnKF with identity map as process model, to avoid confusion with nonlinear effects of obs update and 1d-scatterometer as observation. 40 member ensemble True solution is 0.5 Observations plotted as x-s, but plotting is ambiguous, onloy one possible solution shown. Clearly see that some ensemble members are moved away from the observations. E.g. member at x=-2 at beginning. K changes sign at next update. Ultimately the filter completely ignores the observations, because the cross- covariance of the ensemble goes to zero due to cancellation of positive and negative ensemble members. Cross- covariance =

Implementation matters! Biased EnKF algorithms Work with David Livings and Nancy Nichols

=

Results with ETKF (old formulation) and Peter Lynch’s swinging spring model N=10, Perfect observations Red ensemble mean Blue ensemble std. Error bars indicate obs std. Ensemble statistics not consistent with the truth!

Bias and the EnKF Many EnKF algorithms, can be put into a “square root” framework. Define an ensemble perturbation matrix: So, by definition of the ensemble mean

Square-root ensemble updates The mean of the ensemble is updated separately. Ensemble perturbations are updated as where T is a (non-unique) square root of an update equation. Thus, for consistency, David discovered that not all implementations preserve this property. We have now found nec. and suff. conditions for consistency.

Consequences The size of the ensemble spread will be too small The ensemble will be biased The size of the ensemble spread will be too small Filter divergence is more likely to occur ! Care must be taken in algorithm choice and implementation

Summary and conclusions Data assimilation is invaluable in many applications Care must be taken in the way we apply in Most schemes are loosely based on the Kalman filter that assumes Linear models (M and H) Gaussian statistics Conditional independence between obs Often additionally obs errors assumed uncorrelated Need to at least stop and think about violating assumptions Implementation (numerical analysis) matters!