1. Data Assimilation in Meteorology Chris Budd Joint work with Chiara Piccolo, Mike Cullen (Met Office) Melina Freitag, Phil Browne, Emily Walsh, Nathan Smith and Sian Jenkins (Bath)

2. Understanding and forecasting the weather is essential to the future of planet earth and maths place a central role in doing this Accurate weather forecasting is a mixture of Careful modelling of the complex physics of the ocean and atmosphere Accurate computations on these models Systematic collection of data A fusion of data and computation Data assimilation is the optimal way of combining a complex model with uncertain data

3. Observations from space Upper-air observationsWeather radar Surface observations Integrated forecasting process NWP forecasts AnalysisIntervention Fine-tuning Forecast products and guidance

4. Modelling the global atmosphere and ocean Vertical exchange between layers of momentum, heat and moisture Horizontal exchange between columns of momentum, heat and moisture Vertical exchange between layers of momentum, heat and salts by diffusion, convection and upwelling Orography, vegetation and surface characteristics included at surface on each grid box Vertical exchange between layers by diffusion and advection 15° W 60° N 3.75° 2.5° 11.25° E 47.5° N

5. Met Office Current Configurations North Atlantic and Europe (NAE) Met. Office Global/Regional Ensemble Prediction System (MOGREPS) became fully operational in Sep 2008 after 3 years of trials. Focus on short-range out to 72hr. UK Global 25km L70 model (was 40km L50) Incremental 4D-Var Data Assimilation 60km 24m ETKF ensemble (was 90km L38) Regional NAE 12km L70 model Incremental 4D-Var DA 16km 24m L70 ETKF ensemble (was 24km L38) UK 1.5km model (stretched) (was 4km) Incremental 3D-Var DA

6. Data: Sources of observation

7. Observation Volumes in 6 hours Category Count% used Category Count% use d TEMPs 63799% Satwinds: JMA 261034% PILOTs 30799% Satwinds: NESDIS 1424783% Wind Profiler 135539% Satwinds: EUMETSAT 2209571% Land Synops 1655199% Scatwinds: Seawinds 4365661% Ships 303484% Scatwinds: ERS 270752% Buoys 872763% Scatwinds: ASCAT 2416264% Amdars 6414723% SSMI/S 5321401% Aireps 714412% SSMI 6980481% GPS-RO 77699% ATOVS 11272243% AIRS 758246% IASI 802803%

8. Performance Improvements Met Office RMS surface pressure error over the N. Atlantic & W. Europe “Improved by about a day per decade” Andrew Lorenc DA Introduced

9. What are the causes of improvements to NWP systems? 1.Model improvements, especially resolution and sub grid modelling 2. Better observations 3. Careful use of forecast & observations, allowing for their information content and errors. Achieved by variational data assimilation e.g. of satellite radiances.

10. Basic Idea of Data Assimilation Numerical Weather Prediction NWP calculation gives a predicted state with an error True state of the weather is Make a series of observations y of some function of the true state Eg. Limited set of temperature measurements with error Now combine the prediction with the observations

11. Both the NWP prediction and the data have errors. Can we optimally estimate the atmospheric state which is consistent with both the prediction and the data and estimate the resulting error? NOTE: Approximately 10^9 degrees of freedom 10^6 data points So significantly underdetermined problem

12. Assume initially: 1.Errors are unbiased Gaussian variables 2.Data and NWP prediction errors are uncorrelated 3. H(x) is a linear operator y Best state estimate analysis Use to produce forecast 6 hours later Data NWP prediction

13. Can estimate using Bayesian analyis: Maximum likelihood estimate of data y given Best RMS unbiased estimate of the true state: BLUE Minimum error variance Likelihood Prior Posterior

14. Assumptions about the error Data error: Gaussian, Covariance R Background (NWP) error: Gaussian, Covariance B BLUE: Find which minimises NOTE:

15. If R and B are known then the best estimate of the analysis is Kalman filter: Continuously updates the forecast and its error given the incoming data. Covariance of the analysis error

16. Implementation: In the context of minimising the functional This is implemented as 3D-VAR (since 1999 in the Met Office) : Background, derived from 6 hour NWP forecast : Analysis : NWP forecast using as initial data

17. Ensemble Kalman Filter EnKF This is a widely used Monte Carlo method that uses an ensemble of forecasts to estimate the terms in the Kalman filter Idea: Take a large number of initial states and estimate the resulting background states Estimate

18. y Basic Filtering Idea Advantages: Works well with high dimensional systems Disadvantage: EnKF inaccurate with strong nonlinearity eg. Shear flow [Jones]

19. Corrected forecast Previous forecast 4D VAR … Preferred variational method Use window of several observations over 6 hours Assimilation Window 9 UTC12 UTC15 UTC Obs. x Time xbxb xaxa JoJo Obs. JoJo JbJb JoJo JoJo

20. 4D-VAR idea: Evolutionary model M (nonlinear) Unknown initial state Times Over a time window Leads to state estimates Data yi over window Find so that the estimates fit the data Smoothing

21. Minimise Subject to the strong model constraint At present assume perfect model, but can also deal with certain types of model error (both random and systematic) by using a weak constraint instead

22. Usually solved by introducing Lagrange multipliers And solving the adjoint problems

23. Solution: Find to minimise nonlinear function J Need forward calculation to find and backward solve to find VERY expensive for high dimensional problems!!! Only have limited time to do the calculation (20 mins) Incremental 4D-Var: Cheaper! 1. Assume is close to 2. Linearise J about and minimise this function using an iterative method eg. BFGS 3. Repeat if needed (not usually) BUT: Relies on assumption of near linearity to work well

24. Very effective method!! Met Office operational in 2004 [Lorenc, …. ] Used by many other centres

25. Estimation of the background and covariance errors Good estimates of the covariance matrices R and B are important to the effectiveness of 4D-VAR 1.To get the physics correct 2.To avoid spurious correlations between parameters 3.To give well conditioned systems NOTE: B is a very large matrix, difficult to store and very difficult to update. Impractical to calculate using the Fokker-Plank equation

26. R Different instrument error characteristics and errors of representativeness B Enormous: 10^8 x 10^8 Deduce structure from: Historical data Known dynamical and physical structure of the atmosphere eg. Balance relationships [Bannister]

27. Build meteorology into the calculation of B through Control Variable Transformations (CVTs) IDEA: Choose more ‘natural’ physical variables which have uncorrelated errors so that the transformed covariance matrix is block diagonal or even the identity Set Reduces the complexity of the system AND gives better conditioning for the linear systems

28. Reduces vertical correlations by projecting onto empirical orthogonal vertical modes Separates physical parameters into uncorrelated ones eg. temperature, wind, balanced and unbalanced Effective, but errors arise due to lack of resolution of physical features leading to spurious correlations. Reduces horizontal correlations by projecting onto spherical harmonics

29. Eg. Problems with stable boundary and inversion layers and assimilating radiosonde data Poor resolution leads to inaccurate predictions of fog and ice

31. Solution one : increase global resolution VERY EXPENSIVE!!! Solution two : locally redistribute the computational mesh to resolve the features Cheap and effective! [Piccolo, Cullen, B,Browne, Walsh]

32. Adjust the vertical coordinates to concentrate points close to the inversion layer and reduce correlations Introduce an extra transformation Adaptive mesh transformation applied to latitude-longitude coordinates

33. Do this by using tools from adaptive mesh generation methods for PDES Set: z original height variable new ‘computational’ height variable Relate these via the equation M called the ‘monitor function’ [B, Huang, Russell, Walsh]

34. Take M large if there is active meteorology Eg. High potential vorticity Initially use background state estimate, then update

35. © Crown copyright Met Office Monitor function and the Adaptive Grid Piccolo&Cullen QJR Met Soc 2011

36. © Crown copyright Met Office First calculation UK4 domain: 3 Jan 2011 00z Updated calculation

37. Applied to the Met Office UK4 model RMST (K)RH (%)u (m/ s)v (m/s) Control0.760.0451.321.16 Test0.640.0451.291.16 N obs 1011901819 Test case: 8th Feb 2010. Significant reduction in RMS error especially for temperatures Piccolo&Cullen, QJR Met Soc 2011 Particularly effective for the 2m temperatures

38. © Crown copyright Met Office RMS error: Analysis - Observations thetazonal wind relative humiditymeridional wind

39. Used together with Met Office Open Road software to advise councils on road gritting over Christmas

40. Adaptive mesh implemented operationally in November 2010. Now extending it to a fully three dimensional implementation [B,Browne,Piccolo]

41. Other refinements to 4D-Var Change in the background norm [Freitag, B, Nichols] Total variation: Gives significantly better resolution of fronts, shocks and other localised features But.. Hard to implement in high dimensions!!

42. Dealing with model error If model has random errors with Covariance C can extend 4D-Var to find the minimiser of However, most model errors eg. Numerical errors are systematic. Dealing with these and quantifying the uncertainty in DA is an area of active research [Jenkins, B, Smith, Freitag], [Cullen&Piccolo], [Stuart]

43. Dealing with nonlinearity Better use of appropriate (eg. Lagrangian) data Tuning method to data [Jones, Stuart, Apte] Use of particle filters and MCMC methods [Peter Van Leeuwan] Lot of research into finding a compromise between dealing with the high dimensionality and nonlinearity in the system

44. Conclusions Data assimilation is an optimal way of merging models with data Useful for model tuning, validation, evaluation, uncertainty quantification and reduction Very effective in meteorology Many other applications to Planet Earth eg. Climate change, oil reservoir modelling, geophysics, energy management and even crowd behaviour

45. Problems with the simple Kalman Filter Assumption of Gaussian random variables Assumption of linearity Assumption of known covariances Covariance matrix B is VERY large for meteorological problems Minimisation is a large complex problem

46. Problems with 4D Var Accuracy Estimation of the background covariance matrix Ill conditioning of the linear systems Reliance of near linearity Inappropriate use of background data Incorrect covariance between data Unresolved random and systematic model error Poor resolution in the model Improvements subject to much research