1. © Crown copyright Met Office Successes and Challenges in 4D-Var Third THORPEX International Science Symposium Andrew Lorenc, Monterey, Sept 2009.

2. © Crown copyright Met Office Andrew Lorenc 2 Successes and Challenges in 4D-Var 1.Success: 4D-Var has made a significant contribution to the “day per decade” improvement in NWP skill over my career, but it comes behind model improvements (esp. resolution) and statistical allowance for errors (VAR). 2.Current challenge: getting full information from remote sensing (time-sequences of high-resolution images) is a multi-scale, nonlinear problem. 4D-Var can tackle this. 3.Longer-term challenge: the atmosphere is nonlinear, with an attractor of recognisable “meteorological” features, and non-Gaussian PDFs. But NWP models are so large that only quasi-linear data assimilation methods are affordable. Perhaps an ensemble of spun-up 4D-Vars can help?

3. © Crown copyright Met Office Andrew Lorenc 3 1. Success: Causes of improvements to NWP NWP systems are improving by 1 day of predictive skill per decade. This has been due to: 1.Model improvements, especially resolution. 2.Careful use of forecast & observations, allowing for their information content and errors. Achieved by variational assimilation e.g. of satellite radiances. 3.4D-Var. 4.Better observations.

4. © Crown copyright Met Office Andrew Lorenc 4 Performance Improvements Met Office RMS surface pressure error over the N. Atlantic & W. Europe “Improved by about a day per decade”

5. © Crown copyright Met Office Andrew Lorenc 5 60 Years of Met Office Computers LEO Mercury LEO 1 KDF 9 IBM 360 Cyber 205 Cray YMP Cray C90 Cray T3E NEC SX6/8 IBM Power -Phase 1&2 Moore’s Law 18month doubling time

6. © Crown copyright Met Office Andrew Lorenc 6 Ratio of global computer costs: 1 day’s DA (total incl. FC) / 1 day’s forecast. Only 0.04% of the Moore’s Law increase over this time went into improved DA algorithms, rather than improved resolution! 1 day of MOGREPS (24 member LETKF) / 1 day’s forecast : 56. 1 day of MOGREPS / 1 day’s ensemble: 2.3

7. © Crown copyright Met Office Andrew Lorenc 7 Relative scores 2003-5 + dates of 4D-Var implementation 4D-Var implementation

8. © Crown copyright Met Office Andrew Lorenc 8 CBS N-hem Pmsl T+24 RMSE v analysis Rectangles show 12-month running mean impact period of 4D-Var implementation

9. © Crown copyright Met Office Andrew Lorenc 9 CBS N-hem Pmsl T+72 RMSE v analysis

10. © Crown copyright Met Office Andrew Lorenc 10 CBS S-hem Pmsl T+24 RMSE v analysis

11. © Crown copyright Met Office Andrew Lorenc 11 CBS S-hem Pmsl T+72 RMSE v analysis

12. © Crown copyright Met Office Andrew Lorenc 12 S.Hem. Z500 T+24 rms v analyses 4D-Var 3DVar+ATOVS ATOVS Model+Cov Radiances+Cov NOAA16+AMSU-B FGAT+Cov 2nd ATOVS New stats 12hr 4D-Var Higher res.

13. © Crown copyright Met Office Andrew Lorenc 13 N.Hem. Z500 T+24 rms v analyses 4D-Var 3DVar+ATOVS ATOVS Model+Cov Radiances+Cov FGAT+Cov NOAA16+AMSU-B 12hr 4D-Var New stats 2nd ATOVS Higher res.

14. © Crown copyright Met Office Andrew Lorenc 14 Recent improvements are not simply due to better observations Whole observing systems only give up to 6 hours improvement in skill (Fourth WMO Workshop on the Impact of Various Observing Systems on NWP). “No satellite” OSEs now give better forecasts than “All Obs” OSEs did 6 years ago (Richard Dumelow).

15. © Crown copyright Met Office Andrew Lorenc 15 Impact of different observing systems. Current contributions of parts of the existing observing system to the large-scale forecast skill at short and medium- range. The green colour means the impact is mainly on the mass and wind field. The blue colour means the impact is mainly on humidity field. The contribution is primarily measured on large-scale upper-air fields. The red horizontal bars give an indication of the spread of results among the different impact studies so far available. Fourth WMO Workshop on the Impact of Various Observing Systems on NWP. Geneva, Switzerland, 19-21 May 2008

16. © Crown copyright Met Office Andrew Lorenc 16 Change in OSE results 2001-2007. N-hem 500hPa height ACC. Not the same period, so only make qualitative comparisons! Richard Dumelow

17. © Crown copyright Met Office Andrew Lorenc 17 Some current issues  Trend to higher resolution – multi-scale DA: Global NWP continues to demonstrate benefit as grid decreases to 25km or better. UK NWP benefitting from grid 1.5km or better. Significant DA uncertainty at larger scales can be reduced by more accurate treatment of small scales.  Dense but incomplete remote sensing + dynamical coupling to poorly observed variables through time-history: Global satellite soundings Radar “Multi-resolution incremental 4D-Var with outer-loop” can tackle these.

18. © Crown copyright Met Office Andrew Lorenc 18 2. Current challenge: Multi-scale assimilation of image sequences Getting information from the perceived movement of a detailed tracer field is a multi-scale nonlinear problem. Incremental 4D-Var with an outer-loop can tackle it, at a cost which is becoming affordable.

19. © Crown copyright Met Office Andrew Lorenc 19 Statistical, incremental 4D-Var Statistical 4D-Var approximates entire PDF by a Gaussian. Adjoint of PF model is needed N.B. PF model need not be tangent-linear to full model and in all NWP implementions, is not.

20. © Crown copyright Met Office Andrew Lorenc 20 Perturbation Forecast model for Incremental 4D-Var Cloud fraction (RH total -1)/(1-RH crit ) Minimise: Designed to give best fit for finite perturbations Not Tangent-Linear Filters unpredictable scales and rounds IF tests Requires physical insight – not just automatic differentiation Tim Payne cloud fraction

21. © Crown copyright Met Office Andrew Lorenc 21 Incremental 4D-Var with Outer Loop xgxg xbxb δxδx +δη+δη + η Optional model error terms

22. © Crown copyright Met Office Andrew Lorenc 22 What spread to assume in regularisation? If guess=background, need to approximate whole of PDF f In final outer-loop, only need to approximate PDF a

23. © Crown copyright Met Office Andrew Lorenc 23 Information content of imagery sequences Humans can make reasonable forecasts based on imagery alone (satellite or radar): information scarcely used in NWP. Time-sequences aid the interpretation of images. Some important information is multi-scale; details at high-resolution are used to recognise patterns whose larger-scale movements are significant.

24. © Crown copyright Met Office Andrew Lorenc 24 Current methods for assimilating imagery AMVs (aka cloud track winds) produced from the motion of patterns seen in ~32 2 pixels. Satellite sounders give course-grained imagery repeated every ~6 hours. Met Office recently implemented 4D-Var assimilation of cloud. Radar radial winds and reflectivity are assimilated in research EnKF and 4D-Var systems.

25. © Crown copyright Met Office Andrew Lorenc 25 AMVs I am not suggesting we could replace AMVs by 4DDA in the near future! However they provide an example of demonstrated useful information from imagery sequences, which a method should in principle be able to extract. 4DDA methods could, in theory, improve on current AMV techniques in allowing for development and dynamical coupling of features.

26. © Crown copyright Met Office Andrew Lorenc 26 Comparison of observed and modelled cloud 9Z 13-10-2002 ObservedSimulated Samatha Pullen

27. © Crown copyright Met Office Andrew Lorenc 27 12Z 13-10-2002

28. © Crown copyright Met Office Andrew Lorenc 28 15Z 13-10-2002

29. © Crown copyright Met Office Andrew Lorenc 29 18Z 13-10-2002

30. © Crown copyright Met Office Andrew Lorenc 30 21Z 13-10-2002

31. © Crown copyright Met Office Andrew Lorenc 31 0Z 14-10-2002

32. © Crown copyright Met Office Andrew Lorenc 32 3Z 14-10-2002

33. © Crown copyright Met Office Andrew Lorenc 33 6Z 14-10-2002

34. © Crown copyright Met Office Andrew Lorenc 34 9Z 14-10-2002

35. © Crown copyright Met Office Andrew Lorenc 35 12Z 14-10-2002

36. © Crown copyright Met Office Andrew Lorenc 36 15Z 14-10-2002

37. © Crown copyright Met Office Andrew Lorenc 37 18Z 14-10-2002

38. © Crown copyright Met Office Andrew Lorenc 38 21Z 14-10-2002

39. © Crown copyright Met Office Andrew Lorenc 39 Equations for tracer advection In the linearised equations, changes to the wind depend on the gradient of the linearisation state m, biases in observations or model S′ can change the wind. Determining u & m simultaneously is a nonlinear problem.

40. © Crown copyright Met Office Andrew Lorenc 40 4D-Var applied to imagery sequences

41. © Crown copyright Met Office Andrew Lorenc 41 Nonlinear 4D-Var Lorenc 1988 showed that nonlinear 4D-Var of tracer obs at two times in a shallow water model improved forecast. 4D-Var of tracer at two times 3D-Var of tracer at one time Cycled 3D-Var of tracer at two times Forecast from background

42. © Crown copyright Met Office Andrew Lorenc 42 4D-Var “retrieved” winds T42L19, 24hr, adiabatic, not incremental, no J b

43. © Crown copyright Met Office Andrew Lorenc 43 4D-Var “retrieved” winds QJ 1994

44. © Crown copyright Met Office Andrew Lorenc 44 Linearized Extended Kalman Filter Daley (1995, 1996) studied linearized equations in EKF. Wind field can be recovered provided: sufficient structure in the constituent field, observations are frequent and accurate, data voids are small. i.e. filter estimated field must stay close enough to the truth for gradients to be accurate.

45. © Crown copyright Met Office Andrew Lorenc 45 Equations for tracer advection In the linearised equations, changes to the wind depend on the gradient of the linearisation state m, biases in observations or model S′ can change the wind. Determining u & m simultaneously is a nonlinear problem.

46. © Crown copyright Met Office Andrew Lorenc 46 Will linear incremental 4D-Var work? Wind increments are calculated using gradients of the guess. In a long window (several ob times): Cannot alter both the initial m (to fit early obs) and the wind u which advects it (to fit late obs). the guess is less likely to be accurate. In a short-window cycle (mimicking EKF): u′ will be derived from the movement of background m to fit observations. But 4D-Var does not know in which areas background m is unreliable (due to past data voids) and may derive unreliable u′. Not very well!

47. © Crown copyright Met Office Andrew Lorenc 47 Multi-scale DA If displacement (between obs)  size of features (or if features have sharp edged, e.g. cloud/no cloud): Multiple maxima in fit to obs are possible; Linearisation fails if obs increments fall in regions with zero gradient; So we need a good guess at the displacement. Might obtain this from a preliminary iteration at reduced resolution (such that features are smoothed). This fits well with multiple outer-loop 4D-Var.

48. © Crown copyright Met Office Andrew Lorenc 48 3. Longer-term challenge: Nonlinearity, attractor, non-Gaussianity. The atmosphere is nonlinear, but the best NWP models are so large that only quasi-linear quasi-Gaussian assimilation methods are affordable. Nonlinearity helps: Without it small scale perturbations would grow rapidly and we would be swept away! Coherent, predictable features like inversions, fronts, cyclones are maintained by nonlinear processes. Current NWP is already non-Gaussian: the ensemble-mean “best estimate” is not a plausible meteorological state – it lacks small scales and give a poor precipitation forecast. In practice an ensemble is needed to represent the correct power and uncertainty in small scales. Theoretically, a particle filter can solve the nonlinear non-Gaussian assimilation problem, for a perfect model with the correct attractor. But it is completely unaffordable for NWP. Linear Kalman filter methods cannot constrain states to a nonlinearly defined attractor. But nonlinear 4D-Var using an outer-loop and a long time-window might do so, via an additional constraint that the analysis must be near a spun- up, balanced model state.

49. © Crown copyright Met Office Error growth v scale Growth of errors initially confined to smallest scales, according to a theoretical model Lorenz (1984). Horizontal scales are on the bottom, and the upper curve is the full atmospheric motion spectrum. (from Tribbia & Baumhefner 2004).

50. © Crown copyright Met Office Limits to deterministic 4D-Var with turbulence model Tanguay and Gauthier (1995) showed deterministic 4D-Var does not work for a wide range of scales.

51. © Crown copyright Met Office Andrew Lorenc 51 Lorenc, A. C., 2002: Atmospheric Data Assimilation and Quality Control. Ocean Forecasting, eds Pinardi & Woods. ISBN 3-540-67964-2. 73-96 Posterior PDF when analysing a precise but unreliable observation, and various choices of “best” analysis.

52. © Crown copyright Met Office Andrew Lorenc 52© Crown copyright Met Office Tephigram of sounding, global model background and analysis, for the mean of 136 UK soundings with layer cloud top diagnosed at level 5 in the background.

53. © Crown copyright Met Office Andrew Lorenc 53© Crown copyright Met Office 3. Longer-term challenge: Nonlinearity, attractor, non-Gaussianity. As far as I know (research may prove me wrong!): For the most accurate forecasts and the best assimilation NWP models will resolve detail which we cannot always observe. Linear Gaussian methods will not work. The minimum variance best estimate is not meteorological, and likely to “head off into the bushes”. Full nonlinear methods (e.g. particle filters) are too expensive for NWP. We need simple linear equations to have computationally feasible methods for models with a billion degrees of freedom. We cannot define the “attractor” of meteorological states in practice without relying on an NWP model. (But models will have biases.) Any method must be a compromise, only partially addressing all the above problems. Could try long-window 4D-Var, so that any analysis is close to the model’s attractor and the observations, while unobserved detail is generated by the high- resolution model and stochastic perturbations are used to generate an ensemble to sample this detail.

54. © Crown copyright Met Office Questions and answers

55. © Crown copyright Met Office Andrew Lorenc 55 Successes and Challenges in 4D-Var 1.Success: 4D-Var has made a significant contribution to the “day per decade” improvement in NWP skill over my career, but it comes behind model improvements (esp. resolution) and statistical allowance for errors (VAR). 2.Current challenge: getting full information from remote sensing (time-sequences of high-resolution images) is a multi-scale, nonlinear problem. 4D-Var can tackle this. 3.Longer-term challenge: the atmosphere is nonlinear, with an attractor of recognisable “meteorological” features, and non-Gaussian PDFs. But NWP models are so large that only quasi-linear quasi-Gaussian methods are affordable. Perhaps an ensemble of spun-up 4D-Vars can help?

56. © Crown copyright Met Office Andrew Lorenc 56 1. Success: Causes of improvements to NWP NWP systems are improving by 1 day of predictive skill per decade. This has been due to: 1.Model improvements, especially resolution. 2.Careful use of forecast & observations, allowing for their information content and errors. Achieved by variational assimilation e.g. of satellite radiances. 3.4D-Var. 4.Better observations.

57. © Crown copyright Met Office Andrew Lorenc 57 2. Current challenge: Multi-scale assimilation of image sequences Getting information from the perceived movement of a detailed tracer field is a multi-scale nonlinear problem. Incremental 4D-Var with an outer-loop can tackle it, at a cost which is becoming affordable.

58. © Crown copyright Met Office Andrew Lorenc 58 3. Longer-term challenge: Nonlinearity, attractor, non-Gaussianity. The atmosphere is nonlinear, but the best NWP models are so large that only quasi-linear quasi-Gaussian assimilation methods are affordable. Nonlinearity helps: Without it small scale perturbations would grow rapidly and we would be swept away! Coherent, predictable features like inversions, fronts, cyclones are maintained by nonlinear processes. Current NWP is already non-Gaussian: the ensemble-mean “best estimate” is not a plausible meteorological state – it lacks small scales and give a poor precipitation forecast. In practice an ensemble is needed to represent the correct power and uncertainty in small scales. Theoretically, a particle filter can solve the nonlinear non-Gaussian assimilation problem, for a perfect model with the correct attractor. But it is completely unaffordable for NWP. Linear Kalman filter methods cannot constrain states to a nonlinearly defined attractor. But nonlinear 4D-Var using an outer-loop and a long time-window might do so, via an additional constraint that the analysis must be near a spun- up, balanced model state.