1. , Man Zhang, and Karina Apodaca Warn-on-Forecast and High Impact Weather Workshop, Norman, OK, 8-9 Feb 2012 Challenges of assimilating all-sky satellite radiances Milija Zupanski, Man Zhang, and Karina Apodaca Cooperative Institute for Research in the Atmosphere Colorado State University Fort Collins, Colorado, U. S. A. [ http://www.cira.colostate.edu/projects/ensemble/ ] Acknowledgements: - NOAA NESDIS/GOES-R - NOAA NESDIS/JCSDA - NSF Collaboration in Mathematical Geosciences

2.  Relevance of all-sky radiance assimilation  Current status  Challenges  Future Overview

3. Satellite observations  Remote sensing is the major source of observations - radars - satellites AMSU-A GOES-11 SNDR Need to maximize the utility of cloud observations - hurricanes - severe weather  Satellite data are available everywhere - open ocean - polar regions - other isolated areas on the globe

4. Current status of all-sky radiance data assimilation  Most operational centers assimilate only clear-sky radiances - The wealth of cloud-related measurements is discarded - However, most high impact weather events are characterized by the presence of clouds and precipitation - The consequence is a sub-optimal use of satellite observations  Limited research and operational efforts - Mostly related to the use of variational methods, only recent use of ensemble and hybrid variational-ensemble methods - All-sky microwave data assimilation operational at ECMWF since 2009 (Bauer et al. 2010; Geer et al. 2010 - SSM/I, TMI. AMSR-E) - Pre-operational testing at NCEP, also pursued at other operational weather centers  Potential benefit of all-sky radiance assimilation is generally accepted, but it is difficult to extract the maximum information from these observations - modeling of clouds (e.g., microphysics) - data assimilation methodology - computing resources, high resolution

5. Re-development of the TS Erin (2007): Distribution of AMSU-B radiance data in the NCEP operational data stream: (a) all observations, (b) accepted observations after cloud clearing. Data are collected during the period 15-18Z, August 18, 2007. Note that almost all observations in the area of the storm got rejected by cloud clearing. (from Zupanski et al. 2011, J. Hydrometeorology) Impact of cloud clearing (radiance assimilation) Need assimilation of all-sky radiances to improve the observation information value

6. Motivation - CIRA DA research  Develop a robust and efficient data assimilation for high impact weather events - tropical cyclones - severe weather  Focus on assimilation of cloud and precipitation affected satellite measurements, such as all-sky radiance assimilation  Utilize operational codes as much as possible, focus on realistic issues - WRF NMM, HWRF - GSI - CRTM  Assimilate cloudy radiance from various sources: - microwave, infrared, lightning - combine information from different sources to find most beneficial combinations  List of instruments - New: GOES-R (ABI, GLM), JPSS (ATMS, CrIS), - Existing: AMSU-A,B, MHS, AMSR-E, TMI, MSG SEVIRI, WWLLN

7. Challenges of all-sky radiance data assimilation  Data assimilation: Methodological and computational issues  Microphysical control variables - allow cloud observations to impact hydrometeors  Forecast error covariance - Forecast error covariance needs to be state-dependent, and also to represent dynamical and microphysical correlations  Nonlinearity and non-differentiability of Radiative Transfer (RT) operator  Correlated observation errors  Non-Gaussian errors  Quantifying all-sky radiance information: - How to provide a maximum utility of these data, and how to measure success?  Other relevant issues : verification, code maintenance, bias correction, …  Everything is connected, need to take into account all components

8. Relevance of microphysical control variables Adjustment of microphysical control variables: - provides a more complete control of initial conditions - allows most direct impact of cloud observations on the analysis - critical for high impact weather (e.g., TC and severe weather) Microphysics control variables: impact on DA Physically unrealistic analysis adjustment without hydrometeor control variable (cloud ice in this example) Temperature analysis increment at 850 hPa No cloud ice adjustment > 25 K With cloud ice adjustment 5-10 K

9. Forecast error covariance Analysis correction from variational and ensemble DA can be represented as a linear combination of the forecast error covariance singular vectors u i Singular value decomposition (SVD) of the forecast error covariance Fundamentally important to have adequate forecast error covariance. For clouds and precipitation, this implies flow-dependent and dynamically meaningful representation of model uncertainties. Structure of forecast error covariance defines the analysis correction! The quality of data assimilation can be assessed by examining the structure of forecast error covariance! Use single observation experiment to assess the structure:

10. Forecast error covariance: Algebra Only P dd is well known: -Correlations among microphysical variables not well known -Even less known correlations between dynamical and microphysical variables Complex inter-variable correlations (e.g., standard dynamical variables and microphysical variables) Correlations between dynamical variables Correlations between microphysical variables Cross-correlations between dynamical and microphysical variables

11. Forecast error covariance: Algebra Both methods have limitations in representing cloud-related correlations Variational: modeling of cross-covariances, time-dependence Ensemble: reduced rank Ensemble methods: P RR = reduced rank error covariance Hybrid variational-ensemble DA methods are likely the optimal choice for assimilation of cloud-related observations (i.e. all-sky radiances) Variational methods: P M = modeled error covariance

12. Single observation of cloud snow at 650 hPa: ensemble DA horizontal response -Corresponds to high-frequency MW radiance observation -WRF model (nest at 3km) -09 Sep 2012 at 1800 UTC Horizontal analysis increments for (a) snow, and (b) north-south wind component (b) V-wind at 650 hPa ( P v,snow ) (a) Snow at 650 hPa (P snow,snow ) AB

13. Single observation of cloud snow at 650 hPa: ensemble DA vertical response Vertical analysis increments for (a) snow, and (b) rain. (a) Snow at 34N (P snow,snow ) (b) Rain at 34N (P rain,snow ) Difficult to model rain-snow correlation: non-centered response and time-dependence X

14.  The same cost function can be defined for variational and for Kalman Filter (e.g., EnKF) methods (Jazwinski 1970): - KF: an explicit minimizing solution of quadratic cost function using Newton method - VAR: an iterative solution of an arbitrary nonlinear cost function Nonlinear observation operators: (Forgotten) Role of Hessian preconditioning  Nonlinearities increase for precipitation affected radiances - scattering - clouds, aerosol Cost function in (a) physical, and (b) preconditioned space Hessian preconditioning has to be “balanced” (i.e. similar adjustment in all variables). Otherwise, minimization will create imbalances.

15. Non-differentiable RT observation operators  All-sky radiative transfer calculation has two computational branches: - clear-sky - cloudy and precipitation-affected  Decision about required calculation depends on model variables, thus creates a discontinuity in gradient and/or cost function  Since commonly used iterative minimization is gradient-based, non-differentiability could have a large impact on the analysis Assimilation of all-sky radiances may benefit from non-differentiable minimization, or other means of addressing discontinuities

16. all-sky radiance observation information content (Degrees of Freedom for Signal – DFS) MW: AMSR-E all-sky radiance data assimilation (Erin, 2007) (from Zupanski et al. 2011, J. Hydrometeorology) OBS 89v GHz Tb Wind analysis uncertainty (500 hPa) Degrees of Freedom for Signal (DFS) IR: Assimilation of synthetic GOES-R ABI (10.35 mm) all-sky radiances (Kyrill, 2007) (from Zupanski et al. 2011, Int. J. Remote Sensing) Cloud ice analysis uncertainyDegrees of Freedom for Signal (DFS) METEOSAT Imagery valid at 19:12 UTC 18 Jan 2007 Analysis uncertainty and DFS are flow-dependent, largest DFS in cloudy areas of the storm.

17. Quantification of Shannon information for all-sky radiances Use mutual information (I) and entropy (H) Joint entropy H(Y 1,Y 2 ) can be used to quantify the loss of information due to correlations between all-sky radiance observations Y 1 andY 2 Mutual information of dependent variables is smaller than mutual information of independent variables By definition Use To obtain All-sky radiance observations are correlated How to measure information from correlated observations?

18. Future  Important to develop capability to extract maximum information from cloudy and precipitation-affected radiances  Critical for improved analysis and prediction of TC and severe weather outbreaks  Take into account all relevant components (e.g., current challenges) - partial solutions will not bring true progress - if Gaussian error statistics is not correct, but used, the cost function is inadequate, implying incorrect minimizing solution - unbalanced Hessian preconditioning will adversely change the adjustment of variables by creating dynamical imbalances of the analysis  Computation - RT computation increases 2-3 times with scattering - number of observations increases by an order of magnitude due to cloudy information - 20-30 times more expensive to compute  Improved information measures for all-sky radiances (e.g., assessment)  Combine information from various sources: GOES-R, JPSS (MW, IR, Lightning)

19. References: Non-differentiable minimization Steward, J. L., I. M. Navon, M. Zupanski, and N. Karmitsa, 2011: Impact of Non-Smooth Observation Operators on Variational and Sequential Data Assimilation for a Limited-Area Shallow-Water Equation Model. Quart. J. Roy. Meteorol. Soc., DOI: 10.1002/qj.935. All-sky IR Zupanski D., M. Zupanski, L. D. Grasso, R. Brummer, I. Jankov, D. Lindsey and M. Sengupta, and M. DeMaria, 2011: Assimilating synthetic GOES-R radiances in cloudy conditions using an ensemble-based method. Int. J. Remote Sensing, 32, 9637-9659. All-sky MW Zupanski, D., S. Q. Zhang, M. Zupanski, A. Y. Hou, and S. H. Cheung, 2011: A prototype WRF-based ensemble data assimilation system for downscaling satellite precipitation observations. J. Hydromet., 12, 118-134.