1. Page 1 COST/ESF School: UTLS, Cargese, 3-15 October 2005 DA 12: Evaluation of future missions Author: W.A. Lahoz Data Assimilation Research Centre, University of Reading RG6 6BB, UK

2. Page 2 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Why is it important to evaluate expensive future missions such as ESA’s Envisat Quantify added value from new observations in comparison to Global Observing System Use of data assimilation: Different approaches to evaluating future missions. Observing System Simulation Experiments (OSSEs) and Observing System Experiments (OSEs) Example of an OSSE: The proposed SWIFT instrument, measuring stratospheric winds and ozone Topics:

3. Page 3 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Importance of evaluating future EO missions

4. Page 4 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Envisat: http://envisat.esa.inthttp://envisat.esa.int

5. Page 5 COST/ESF School: UTLS, Cargese, 3-15 October 2005 You are given 2.3 BEuros for Envisat to observe the Earth System… Hooray! BUT… How can you check if this is a good use of money? How can you quantify value?

6. Page 6 COST/ESF School: UTLS, Cargese, 3-15 October 2005 What do you need to consider? NOT the value of Envisat BUT added value of Envisat above what else will be available -> INCREMENTAL VALUE THIS IS TRUE FOR ANY ADDITION TO GOS

7. Page 7 COST/ESF School: UTLS, Cargese, 3-15 October 2005 So… What will the GOS be like? Existing & planned satellite missions What type of observations do we include? Conventional: ground-based, sondes, aircraft Satellites: operational, research

8. Page 8 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Global Earth Observing system for 2008-2010

9. Page 9 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Observation types used by Met Office for NWP

10. Page 10 COST/ESF School: UTLS, Cargese, 3-15 October 2005 GOS: Observing characteristics Viewing geometries (limb/nadir; LEO/GEO) Sonde & ground-based observations distribution Aircraft corridors Observation errors: random, systematic, representativeness See DA 11

11. Page 11 COST/ESF School: UTLS, Cargese, 3-15 October 2005 We have a description of future GOS BUT… How do we get values of a future instrument: obs & errors? Need to sample the “truth” (nature) -> Get from a model run, or analyses

12. Page 12 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Lack of global observations of stratospheric winds in the current operational meteorological system: No sondes above 10 hPa (no global coverage anyway) AMVs from satellites in troposphere Wind information from temperature nadir sounders in extra-tropics (troposphere/stratosphere) But, thermal wind relation breaks down in tropics We have no good current estimates of state of the tropical stratosphere: Variability in the quasi-biennial oscillation (QBO) is underestimated “Balanced” winds problematic for estimating varability of QBO Randel et al. (2002) A current concern about GOS are winds

13. Page 13 COST/ESF School: UTLS, Cargese, 3-15 October 2005 “Realistic” quasi-biennial oscillations in the MO Unified Model MO observational analyses of equatorial winds for Nov 1992 - Jan 2000

14. Page 14 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Recent past: UARS WINDII: mesospheric winds UARS HRDI: stratospheric winds, but impact marginal as observed winds not accurate enough compared to forecasts (Boorman et al. 2000) Future: ESA ADM-Aeolus: launch October 2007 CSA SWIFT: launch 2010? Missions measuring winds

15. Page 15 COST/ESF School: UTLS, Cargese, 3-15 October 2005 ADM-Aeolus Doppler Wind Lidar (DWL) 1 component global wind profiles up to ~30 km N.B. need DA to get 2 components Better information to predict weather Global wind profiles for the entire planet, including remote areas lacking any g-based weather station Main objective: Correct major deficiency in winds in current GOS Increased skill in NWP Data needed to address WCRP key concerns: Quantification of climate variability Validation & improvement of climate models Process studies relevant to climate change OSSEs done for ADM-Aeolus (e.g. Stoffelen)

16. Page 16 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Use of data assimilation

17. Page 17 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Philosophy: Nature, Truth (T) -> sample to get all observations Test impact of one observation (e.g. Envisat-like) Control experiment (C): all observations less that of interest Perturbation experiment (P): all observations Compare C & P against T QUANTIFY!

18. Page 18 COST/ESF School: UTLS, Cargese, 3-15 October 2005  Space agencies (ESA, NASA, JAXA, CSA) invest a lot of money on missions (e.g. ESA’s Envisat has cost 2.3BEuros)  Important to evaluate beforehand possible benefits of future missions, especially those involving satellites Techniques for evaluating future missions:  A technique often used by the space agencies is the OSSE  Information content (e.g. Prunet et al. QJ 98, IASI)  Ensembles (e.g. Andersson et al. WMO 2005, ADM-Aeolus)  OSSEs tend not be used as much by the met agencies  This is due to the shortcomings of OSSEs (see later) -> less attractive to the met agencies Future EO missions

19. Page 19 COST/ESF School: UTLS, Cargese, 3-15 October 2005  A technique often used by the met agencies to evaluate components of an existing observing system is the “Observing System Experiment” (OSE)  An OSE studies the impact of one observation type by removing it from the system under study - test impact of satellite data for NH/SH - test impact of nadir/limb geometries Note:

20. Page 20 COST/ESF School: UTLS, Cargese, 3-15 October 2005 OSSE goal: evaluate if the difference P-T (measured objectively) is significantly smaller than the difference C-T  Simulated atmosphere (“truth”; T): using a model  Simulated observations of instruments appropriate to the study, including errors: using T  Assimilation system: using a model  Control experiment C: all observations except those under study  Perturbation experiment P: all observations Structure of an OSSE

21. Page 21 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Note shortcomings of an OSSE:  Expensive (cost ~ assimilation system) -> alleviate problem: “reduced OSSE” (e.g. profiles instead of radiances) Note: “reduced OSSE” generally only useful when observation of interest has relatively high impact (e.g. stratospheric winds)  Difficult interpretation (model dependence) -> alleviate problem: conservative errors, several methods to investigate impact  Incest -> alleviate problem: different models to construct “truth” & perform assimilation (BUT there could be bias between models) Despite shortcomings, high cost of EO missions means that OSSEs often make sense to space agencies

22. Page 22 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Example of an OSSE

23. Page 23 COST/ESF School: UTLS, Cargese, 3-15 October 2005 SWIFT: see Lahoz et al. QJ 2005  Based on UARS WINDII principle (Doppler effect)  2 wind components using 2 measurements at ~90 o  Thermal emission (mid-IR) of ozone (1133 cm -1 )  Technology difficult to implement  Global measurements of wind and ozone profiles (~20-40 km) OSSE: evaluate proposed SWIFT instrument

24. Page 24 COST/ESF School: UTLS, Cargese, 3-15 October 2005 1.Current observing system: No operational observations of winds for levels above those of radiosondes (~10 hPa) Note: indirect information on winds can be obtained from nadir soundings of temperature (thermal wind; but this breaks down in the tropics) 2. Science:  Help build climatologies of tropical winds  Transport studies (e.g. ozone fluxes)  Use assimilation to obtain 4-d quality-controlled datasets for scientific studies (e.g. climate change and its attribution) Why SWIFT?

25. Page 25 COST/ESF School: UTLS, Cargese, 3-15 October 2005  Establish basis for assimilating SWIFT observations (u, v; ozone)  Investigate scientific merits of SWIFT observations Models used:  “Truth” (ECMWF directly, or forcing a CTM)  Assimilation system (Met Office) (cf. incest) Simulated observations: Operational: C {MetOP, MSG, sondes, balloons, aircraft, surface} Temperature, winds, humidity, ozone SWIFT; C+SWIFT = P Ozone, winds (stratosphere, conservative errors) Several assimilation experiments; analyses evaluated. Qualitative & quantitative tests Design of SWIFT OSSE

26. Page 26 COST/ESF School: UTLS, Cargese, 3-15 October 2005  SWIFT: N - and S - observations (87°N-53°S, 53°N-87°S): non sun- synchronous orbit  - winds 16-50km, every 2km approximately - ozone 16-44km, every 2km approximately  Errors: conservative; random; representativeness error considered to be relatively unimportant SWIFT characteristics

27. Page 27 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Several tests -> robustness (cf. interpretation of the OSSE)  Qualitative (histograms, monthly means)  Quantitative (RMS statistics, significance tests) Assumption: We can discount the bias between the ECMWF and Met Office models because it is removed when comparing P-T and C-T Same bias  in both P-T and C-T differences which are compared Evaluation of SWIFT impact

28. Page 28 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Y=Abs(C-T) -Abs(P-T); Zonal-wind (m/s); January 2000; Shaded:95% C.L. & Y>0. Similar results for April 2000. 10 hPa 1 hPa Significance tests Areas > 5% N.B. Some areas of -ve impact (new obs can degrade DA system) - not significant

29. Page 29 COST/ESF School: UTLS, Cargese, 3-15 October 2005 SWIFT winds  Significant impact in tropical stratosphere EXCEPT lowermost levels  Can have significant impact in extra-tropics when:  SWIFT observations available  Flow regime is variable (relatively fast changing)  Have scientific merit in that they improve:  Information on tropical winds  Wintertime variability  Useful for forecasting & producing analyses to help study climate change & its attribution: better models, better initial conditions, model evaluation SWIFT ozone  Significant impact at 100 hPa & 10 hPa -> regions of relatively high vertical gradient Conclusions

30. Page 30 COST/ESF School: UTLS, Cargese, 3-15 October 2005 “Reduced OSSE”: Radiances used for AMSU-A, IASI at time of SWIFT launch 1. Expectation is that impact in tropics will not change 2. Impact in the extra-tropics may remain unchanged (impact in flow regimes which change relatively fast) Thermal wind relationship does not hold for 1, and is not accurate in 2 Caveats

31. Page 31 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Higher horizontal resolution -> Less “thinning” of satellite data (reduce number of obs): AMSU-A, IASI  Would impact stratospheric wind analyses in the extra-tropics  Conclusions in tropics should be robust  Conclusions for ozone analyses (100 & 10 hPa) should not change NOTE: the SWIFT OSSE did not evaluate forecasts (only analyses) and did not calibrate the OSSE by, e.g., removing simulated sonde data & replacing it with real sonde data Caveats

32. Page 32 COST/ESF School: UTLS, Cargese, 3-15 October 2005 You might think OSSEs are a load of… (fill in words) If so help is at hand! Information content Ensembles Alternatives

33. Page 33 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Aim: evaluate the potential benefit of future sensors compared to other available sensors Prunet et al. (1998) used approach to quantify impact of information content in simulated IASI radiances vs information content in TOVS radiances. Impact of IASI radiances estimated by comparison of analysed errors (which include TOVS or IASI data) vs those of a background field (from model forecast excluding both TOVS and IASI data). If observation type of interest has positive impact, analysed errors should be smaller than background errors. By comparing errors of analyses including TOVS or IASI data, relative information content in these data can be evaluated, and assessment made of their relative benefit. Information content

34. Page 34 COST/ESF School: UTLS, Cargese, 3-15 October 2005 In principle, information content approach is simpler & less expensive to apply than an OSSE. However, information content approach requires a realistic characterization of background & observation errors, which could be difficult to achieve. Furthermore, it could be argued that OSSE approach provides a more complete test of the future sensors. Information content

35. Page 35 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Ensembles Courtesy Andersson et al. ECMWF

36. Page 36 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Ensembles Courtesy Andersson et al. ECMWF

37. Page 37 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Ensembles Courtesy Andersson et al. ECMWF According to Andersson et al: Need “truth” for simulated data

38. Page 38 COST/ESF School: UTLS, Cargese, 3-15 October 2005 Important to quantify value of future missions -> participation of all actors: multi-disciplinary -> quantify benefits: OSSEs & other methods -> caveats: set up experiments carefully Should there be a dedicated OSSE facility? Way forward: