1. Robin Hogan Julien Delanoë Nicola Pounder University of Reading Synergistic cloud, aerosol and precipitation products Progress so far in RATEC

2. Develop best estimate first Other products follow from this Synergy products as defined in CASPER

3. Overview of our role in RATEC Radiative transfer for EarthCARE (RATEC) –Lead contractor: Howard Barker –Started August 23, 2009 –15 month project Our role is to start the development of synergy algorithms –Also define the required imager and target classification products WP120: Review - first 3 months –Products and Algorithms Requirements Document (PARD) –Algorithm Development Plan (ADP) WP 220: Design - next 5 months –Product Definition Document (PDD) WP 320: Prototype and test - final 7 months –Algorithm Theoretical Basis Document (ATBD)

4. Overview of talk Laying the foundations for the algorithm... Summary of retrieval framework –State variables (describing ice, liquid, rain and aerosol) –Forward models (for radar, HSRL lidar, infrared & solar radiometers) –Cost function Minimization techniques –Gauss-Newton & Levenberg-Marquardt –Gradient Descent (adjoint method) –Ensemble methods Coding progress –C++ library –Scattering code Calculation and storage of error descriptors –Solution error covariance matrix –Averaging kernel Progress for specific atmospheric constituents –Liquid cloud: gradient constraint

5. Retrieval framework Ingredients developed before Not yet developed 1. New ray of data: define state vector Use classification to specify variables describing each species at each gate Ice: extinction coefficient, N 0, lidar extinction-to-backscatter ratio Liquid: extinction coefficient and number concentration Rain: rain rate and mean drop diameter Aerosol: extinction coefficient, particle size and lidar ratio 3a. Radar model Including surface return and multiple scattering 3b. Lidar model Including HSRL channels and multiple scattering 3c. Radiance model Solar and IR channels 4. Compare to observations Check for convergence 6. Iteration method Derive a new state vector 3. Forward model Not converged Converged Proceed to next ray of data 2. Convert state vector to radar-lidar resolution Often the state vector will contain a low resolution description of the profile 5. Convert Jacobian to state-vector resolution Jacobian initially will be at the radar-lidar resolution 7. Calculate retrieval error Error covariances and averaging kernel

6. Proposed state variables State variableRepresentation with height / constraintA-priori Ice clouds and snow Visible extinction coefficientOne variable per pixel with smoothness constraintNone Number conc. parameterCubic spline basis functions with vertical correlationTemperature dependent Lidar extinction-to-backscatter ratioCubic spline basis functions20 sr Riming factorLikely a single value per profile1 Liquid clouds Liquid water contentOne variable per pixel but with gradient constraintNone Droplet number concentrationOne value per liquid layerTemperature dependent Rain Rain rateCubic spline basis functions with flatness constraintNone Normalized number conc. N w One value per profileDependent on whether from melting ice or coallescence Melting-layer thickness scaling factorOne value per profile1 Aerosols Extinction coefficientOne variable per pixel with smoothness constraintNone Lidar extinction-to-backscatter ratioOne value per aerosol layer identifiedClimatological type depending on region Ice clouds largely done in CASPER; Snow & riming in convective clouds needs to be added (VARSY?) Liquid clouds to be tackled in RATEC Basic rain to be added in RATEC; Full representation in VARSY? Basic aerosols added in RATEC; Full representation via collaboration (IRMA?)

7. Forward model components From state vector x to forward modelled observations H(x)... Ice & snowLiquid cloudRainAerosol Ice/radar Liquid/radar Rain/radar Ice/lidar Liquid/lidar Rain/lidar Aerosol/lidar Ice/radiometer Liquid/radiometer Rain/radiometer Aerosol/radiometer Radar scattering profile Lidar scattering profile Radiometer scattering profile Lookup tables to obtain profiles of extinction, scattering & backscatter coefficients, asymmetry factor Sum the contributions from each constituent Gradient of radar measurements with respect to radar inputs Gradient of lidar measurements with respect to lidar inputs Gradient of radiometer measurements with respect to radiometer inputs Jacobian matrix H=y/x Lots of expensive matrix multiplications: likely to be the most expensive part of the entire algorithm x Radar forward modelled obs Lidar forward modelled obs Radiometer forward modelled obs H(x)H(x) Radiative transfer models

8. Solution methods (1/3) We want to minimize this cost function: Possibly using the gradient (a vector): …and the second derivative (a matrix): 1. Gauss-Newton method (Rodgers p85): –Advantage: rapid convergence (instant convergence for linear problems) –Another advantage: get the error covariance of the solution for free –Disadvantage: need the Jacobian of every forward model: can be expensive

9. Solution methods (2/3) 2. Gradient descent method (e.g. ECMWF data assimilation system): –Advantage: we dont need to calculate the Jacobian so forward model is cheaper! –Quasi-Newton method to get search direction (conjugate gradient, BGFS etc) –Disadvantage: more iterations needed since we dont know curvature of J(x) –Disadvantage: dont get the error for free at the end Why dont we need the Jacobian H? –The adjoint of a forward model takes as input the vector {.} and outputs the vector J obs without needing to calculate the matrix H on the way –Adjoint can be coded to be only ~3 times slower than original forward model –Tricky coding for newcomers, although some automatic code generators exist Simple steepest descent requires many steps Conjugate gradient method more intelligent…

10. Solution methods (3/3) 3. Levenberg-Marquardt (hybrid of Gauss-Newton & steepest descent): –If cost function reduces, keep small: rapid convergence –If cost function increases, increase so that always go downhill –More robust than Gauss-Newton, but still need the Jacobian 4. Ensemble method (e.g. Ensemble Kalman Filter): –Create ensemble of trial solutions or particles in state space –Spread of results allows Jacobian to be calculated automatically to create better set of particles: repeat until convergence –Applied to GPM retrievals by Mircea Grecu (NASA) –Advantage: no Jacobian required –Estimate of retrieval error from final spread –Disadvantage: may need many ensemble members so probably slower than adjoint method 5. Simulated annealing (e.g. Donovan in ECSIM) –Suitable for very non-linear problems –Too slow for this application?......

11. Computational cost can scale with number of points describing vertical profile N; we can cope with an N 2 dependence but not N 3 Radiative transfer forward models Radar/lidar modelApplicationsSpeedJacobianAdjoint Single scattering: = exp(-2 ) Radar & lidar, no multiple scatteringNN2N2 N Platts approximation = exp(-2 ) Lidar, ice only, crude multiple scatteringNN2N2 Photon Variance-Covariance (PVC) method (Hogan 2006, 2008) Lidar, ice only, small-angle multiple scattering N or N 2 N2N2 N Time-Dependent Two-Stream (TDTS) method (Hogan and Battaglia 2008) Lidar & radar, wide-angle multiple scatteringN2N2 N3N3 N2N2 Depolarization capability for TDTSLidar & radar depol with multiple scatteringN2N2 N2N2 Radiometer modelApplicationsSpeedJacobianAdjoint RTTOV (used at ECMWF & Met Office)Infrared and microwave radiancesNN Two-stream source function technique (e.g. Delanoe & Hogan 2008) Infrared radiancesNN2N2 RADIANT (from Graeme Stephens)Solar radiancesNN2N2 N Lidar will use PVC+TDTS, radar will use single scattering+TDTS Jacobian of TDTS is too expensive: need to develop reduced-resolution Jacobian or alternatively adjoint code Also need depolarization forward model with multiple scattering Infrared will probably use RTTOV, solar radiances will use RADIANT

12. Coding progress Coding has started on the underpinning libraries –Use C++ because of key language features Matrix and Vector Expression Library –Operator overloading and expression templates allows complex operations on matrices to be written on one line, with similar efficiency as Fortran-90 –Important feature is to allow a matrix object to behave as part of another matrix – important when different parts of the Jacobian refer to different instruments and need to be passed to functions in bits Optimal Synergy Library –Object orientation and polymorphism: Radar and Lidar classes inherit from generic Observation class; IceCloud and Aerosol inherit from generic Constituent class –Enables code to be written to be completely flexible: observations can be added and removed without needing to keep track of indices to matrices, so same code can be applied to many platforms Also written code to provide scattering libraries in NetCDF files

13. Storage of error descriptors Measurements Retrievals Measurements If ice is present at N points and M variables are retrieved: –MN retrieved variables to report with MN errors Error covariance and averaging kernel matrices also needed –A standard output from a variational algorithm –But ~(MN) 2 /2 values in each matrix, much larger than the retrieved variables and their errors –Matrix is a different size each ray! Need a suitable compression strategy...

14. Top -> Height -> bottom S N N Retrieval error covariance matrix (Extinction, lidar ratio, N)

15. Retrieval error correlation matrix (Extinction, lidar ratio, N) Top -> Height -> bottom S N N

16. For autocorrelation of extinction... 3 Gaussians requires 8N or fewer values to approximate the error covariance matrix Need to work on correlations between different variables

17. Height (m) Information content Averaging kernel The averaging kernel matrix expresses how the retrieval at a point depends on the truth at every other point –The sum of each column expresses how much the retrieval at that point derives from the measurements (rather than the prior): 1 is good! –The shape of each column expresses the effective vertical resolution of the retrieval: more peaked is good! Easy to parameterize: –Store integral and peak –Or store integral and the width for each column

18. Gradient constraint We have a good constraint on the gradient of the state variables with height for: –LWC in stratocu (adiabatic profile, particularly near cloud base) –Rain rate (fast falling so little variation with height expected) Not suitable for the usual a priori constraint Solution: add an extra term to the cost function to penalize deviations from gradient c: A. Slingo, S. Nichols and J. Schmetz, Q. J. R. Met. Soc. 1982

19. Example in liquid clouds Using simulated observations: –Triangular cloud observed by a 1- or 2-field-of-view lidar –Retrieval uses Levenberg-Marquardt minimization with Hogan and Battaglia (2008) model for lidar multiple scattering Optical depth=30 Footprint=100m Footprint=600m Two FOVs: very good performance One 10-m footprint: saturates at optical depth=5 One 100-m footprint: multiple scattering helps! One 10-m footprint with gradient constraint: can extrapolate downwards successfully

20. Ongoing work Implement various instrument forward models –Not too difficult: interfacing different codes and languages –Write adjoint for slowest models Solver –Test gradient descent algorithms (conjugate gradient, BGFS etc) Implementation of different atmospheric constituents –Ice cloud: we know how to do this –Liquid cloud: to be done in RATEC –Aerosol and precipitation: basic implementation in RATEC; longer-term development needs collaboration with DAME, IRMA etc... Test datasets –Several airborne comparisons for validation (TC4, POLARCAT etc) –A-Train –ECSIM (after RATEC) Two-instrument synergy products –To be tested using same code removing one instrument (after RATEC)

22. Aerosol refractive index Values from Ellie Highwood Strong dependence on aerosol type Varies with hydration Can we use one continuous variable to represent different aerosol properties? Need to make simplifications if information from multiple wavelengths is to be combined 355-nm lidar Solar MSI channels Thermal MSI channels

23. Matrix and Vector Expression Library Intuitive interface and ability to do complex expressions Matrix M(3,3); Vector x(3) = 1.0, 2.0, 3.0;// Simple initialization M = N * R + exp(P);// * invokes element multiplication y = transpose(x) ()* inverse(R);// ()* invokes matrix multiplication x = solve(A, b);// Linear algebra Important aspects not in any other library (to my knowledge) S.subset(M, 3, 30, 2, 20);// Matrix subsetting M(find(M < 0.0)) = 0.0;// Matlab-style indexing BitfieldMatrix b; b.bit(4) = 1;// Handling of classification data Efficient and robust implementation –Expression templates to avoid temporary objects in long expressions –Reference counting for allocated objects protects against memory leaks

24. Key classes State Methods: solve() add_obs(Observation) Data: List of Observations List of Constituents Vector x, y Matrix Jacobian etc. Observation Methods: calc_scat_props() virtual forward_model() Data: Real wavelength Vector y, y_error Matrix Jacobian Constituent Methods: calc_scat_props(wavelength) Data: Vector x, x_prior ActiveInstrument Methods: forward_model() Lidar Radiometer Methods: forward_model() Radar IceCloud LiquidCloud Aerosol Rain BackgroundConstituents

25. Forward model part 1 x ice. x liq. State vector x ln vis. ln N 0. ln vis. ln N 0 *. x highres,ice / x ice Basis functions for variables represented at reduced resolution = x highres,ice W1W1 x ice radar. radar. x radar,ice / x highres,ice x radar,ice W2W2 Scattering lookup table To radiative transfer radar. radar. 0 radar. radar. radar. radar. radar. x radar / x radar,ice = + + x radar,ice W3W3 x radar,gas x radar,liq x radar Sum the contributions from each atmospheric constituent

26. Forward model part 2 Z...Z... y radar / x radar Radiative transfer y' radar H radar To solver Z. '. I y' y / x H Forward modelled observations and Jacobian y radar /x ice H radar,ice y radar / x radar x radar / x radar,ice x radar,ice / x highres,ice x highres,ice / x ice W3W3 W2W2 W1W1 H radar =