Rutherford Appleton Laboratory OEM retrievals with IASI, AMSU and MHS: Summary of PM2 PM3, Teleconference 12 November 2014 R.Siddans, D. Gerber (RAL),

Purpose of the Study To evaluate the benefit of adding microwave (MW) channels to the measurement vector of Eumetsat’s optimal estimation method (OEM) based scheme for retrieving temperature, humidity and ozone from the infra-red (IR) sounder IASI. Eumetsat provide the description and input data of the baseline (IR-only) OEM scheme which is to be extended in the study. The study should also extend the scheme To fit surface spectral emissivity (IR and MW) To work in the presence of (some) cloud (but not precipitation) Additionally the impact of some specific AMSU channels (reflecting Metop- A performance) is to be studied

Study Tasks / Work Breakdown Schedule KO in December 2013 WP1000 : Definition of the Sy matrix in the microwaves Input via consultancy from Bill Bell (Met Office) WP2000: OEM(MWIR/Metop-B) over ocean, clear sky Set up IASI OEM to match EUMETSAT L2 PPF configuration Run retrievals (IR and MWIR) and analyse residuals WP3000 : OEM(MWIR/Metop-B) over land, clear sky, with fixed emissivities WP4000 : OEM(MWIR/Metop-B) over land, clear sky, with variable emissivities WP5000 : OEM(MWIR/Metop-B) in partial or full cloudy IFOVs WP6000 : Retrievals with one or more missing AMSU channels WP7000 : Delivery of datasets and final reporting Study to conclude beginning of December this year We Are Here

Overview of the Eumetsat IASI scheme OEM solves under-constrained inverse problem using a prior estimates of the parameters to be retrieved, yielding the most likely solution given the characterised errors on the measurements and the prior estimate. This is achieved by solving the cost function: Where x is state vector (parameters to be retrieved) y is measurement vector (a subset of IASI PC re-constructed / filtered radiances), with errors characterised by covariance S y F(x) is forward model which predicts measurements given state (RTTOV v10.2) x a is the a priori state, which is assumed to have error characterised by S a The cost function is minimised using iterative approach (Newtonian) K is the weighting function matrix – derivatives of F(x) with respect to x

Overview of the Eumetsat Scheme: A priori In the Eumetsat OEM, the a priori state (and first guess) is given by separate statistical retrieval which uses selected IASI, AMSU and MHS measurements as predictors to estimate profiles of temperature, humidity and ozone, together with surface temperature. The relationship between the predictors and the state is derived using a the piece-wise linear regression (PWLR) scheme, training 12 days of measurements against co-located ECMWF analyses The state is expressed in terms of principle components of the covariance of the PWLR profiles against the analyses. 28 principle components are used to represent temperature, 18 for humidity and 10 for ozone. The OEM retrieves the weights of each of these profile principle components (+surface temperature) Temperature is retrieved in K, humidity and ozone in ln(ppmv) The use of PWLR as prior, means the prior state is already rather good. The PWLR is also relatively insensitive to cloud (as the empirical regression will handle this to some extent). -> It is a challenge for the OEM to be “better” than PWLR

Task 1: Definition of the Sy matrix for the MW channels Literature review of use of AMSU+MHS in OEM Dr William Bell (Met Office) consultant to consortium to provide expertise on use of AMSU+MHS Met Office will provide estimates of the NEDT for all of the AMSU-A / MHS channels, as well as the observation covariances used in the operational assimilation of ATOVS radiances In addition we estimate statistics (bias and covariances) of the AMSU/MHS departures from the provided IASI OEM and piece-wise linear regression (PWLR) profiles. Currently measurement covariances are based on differences between MW observations and those computed using the IASI OEM retrieved state

Observation – simulations after MW bias correction

Observation coveariance derived from MW residuals from IASI retrieval

Task 2: OEM (MWIR/Metop-B) over ocean, clear-sky Implement the IASI PPF OEM settings in RAL code Apply scheme to selected days of IASI and AMSU/MHS cloud-free data over ocean/land, to generate results for IR only and MW+IR (MWIR). Days selected: 17 April, 17 July, 17 October 2013 (Metop B) Evaluate results using the diagnostics such as: PWLR, OEM(IR), OEM(MWIR) cf reference profiles (ECMWF analysis) vertical profiles of bias, standard deviation histograms and scatter plots for selected pressure levels maps of departures The water-vapour shall analysed in mixing and relative humidity. DOFS, AKs, fit residuals of OEM(IR) cf OEM(MWIR) RTTOV internal emissivity atlases (TELSEM/CNRM/Wisconsin) + sea model used Eumetsat cloud, precipitation and sea-ice masking used Task 3: OEM (MWIR/Metop-B) over land clear-sky

Retrievals run over both sea (T2) and land (T3) All 3 days (17 April, 17 July, 17 October 2013) IR-only retrievals compared to Eumetsat ODV Differences small cf noise and mainly related to different convergence approach, which affects scenes for which final cost high (deserts, sea ice) MWIR retrieval run with 2 options for Sy With / without off-diagonals included Linear simulations also performed for 4 sample scenes to assess information content Additional case of 0.2K NEBT (uncorrelated) Approximate perfect knowledge of MWIR emissivity Tasks 2+3: Initial IR + MWIR retrievals

Summary of DOFS

Summary from Linear Simulations Using the derived observation errors, IASI+MHS add 2 degrees of freedom to temperature and about half a degree of freedom to water vapour. Effects on ozone are negligible. Neglecting off-diagonals reduces DOFS on temperature and water vapour by about 0.1 (a small effect). For temperature, the improvements are related mainly to the stratosphere though some improvement is also noticeable in the troposphere, esp over the ocean (where the assumed measurement covariance is relatively low). For water vapour improvements are mainly related to the upper troposphere, and penetrate to relatively low altitudes in the mid-latitudes. Assuming 0.2 K NEBT errors (ideal case!) to apply to all channels adds an additional degree of freedom to temperature and an additional half a degree of freedom to water vapour, in some cases considerably sharpening the near-surface averaging kernel.

Statistical assessment of retrievals Based on comparing retrieval to analysis (ANA), Eumetsat retrieval (ODV), PWLR and analysis smoothed by averaging kernel (ANA_AK): x’ = a + A ( t - a ) Where a is the a priori profile from the PWLR, t is the supposed "true", A is the retrieval averaging kernel Profiles smoothed/sampled to grid more closely matching expected vertical resolution (than 101 level RTTOV grid), to avoid spurious structures: Temperature: 0, 1, 2, 3, 4, 6, 8, 10, 12, 14, 17, 20, 24, 30, 35,40,50 km. Water vapour: 0, 1, 2, 3,4, 6, 8, 10, 12, 14, 17,20 km Ozone: 0, 6, 12, 18, 24, 30, 40 km. The grid is defined relative to the surface pressure / z*. Profile results further summarised (for maps, tables) into 3 layers BL: 0-2 km z* (above surface) LT: 0-6 km UT: 6-12 km The mean value of individual profiles taken over these ranges, then stats calculated (summarises bias over these layers)

Ret-Ana bias resolved by AK; Ret-Ana similar with/without MW Ret-Ana_AK “degrades with MW) PWL Similarly “degrades” MWIR vs IR Temperature

MWIR vs IR Humidity Ret-Ana stdev. Better with MW (not Ana_AK, but still Ret_Ana_AK “degrades” less than PWLR

Task 4: Addition of surface emissivity to state vector Basic approach Emissivity included in state vector in terms of principle components: Eigenvectors of global covariance from RTTOV atlases (for the 3 days assessed in the study) A priori covariance is diagonal, filled with corresponding Eigenvalues Have co-located spectra for MW+IR so can include correlations between MW and IR in the prior constraint Land and sea (and permanent land ice) spectra included in the covariance, so retrieval should ~work around coast. IR Atlas based on Wisconsin principle components of natural materials. 416 spectral patterns defined (at 416 wavelengths from cm -1 ) but Only leading 6 patterns used (limit from MODIS channels) Spectral shapes of further patterns needed to explain IASI observations, but no measure of their occurrence globally is available to define the prior constraint for these. Other patterns included by deriving residual patterns not explained by RTTOV-atlas based patterns; Shift of mean emissivity also fitted.

Surface emissivity Eigenvectors and values fitted (including MW correlations).

Task 4: Retrieval Simulations Large set of retrievals conducted to asses benefit MW/IR and performance of emissivity retrieval: standard: IR only, RAL retrieval ~ EUMETSAT OEM. IR+MW: IR+MW retrieval (no emissivity, no cloud retrieval). MW: MW only retrieval (no emissivity, no cloud retrieval). IR+MW; Cloud: IR+MW retrieval with cloud fraction and height also retrieved. Emis:[10/20/30]n: IR only retrieval, with 10/20/30 spectral emissivity patterns retrieved (no emissivity correlations between IR and MW). IR+MW; Emis:20: IR+MW retrieval, with 20 spectral emissivity patterns retrieved. Spectral correlations assumed between IR and MW. IR+MW; Emis:20n: As above, no spectral correlations IR/MW MW; Emis:20: MW only retrieval, with 20 spectral emissivity patterns IR+MW; Emis:20; Cloud: As above, also with cloud retrieved Two versions of each; with PWLR as a priori and a new “climatological constraint”

Cost function + Number of iterations: MWIR with emissivity

Maps of retrieved emissivity spectral pattern coefs

Fitting scale factors for residual patterns High cost over ice surface reduced by fitting scale factors for bias correction (mean + x-track dependence) Causes bias correction to be suppressed over cold surfaces Original schemeFitted emissivityFitted emissivity Fitted Bias Correction

All scenes Cloud-free

Testing emissivity retrieval (desert) MWIR Emis = RTTOV

Testing emissivity retrieval (desert) MWIR Emis a priori = RTTOV MWIR Emis a priori = 1 Climatological prior

Testing emissivity retrieval (desert) IR Emis a priori = 1;ret bias correction MWIR Emis a priori = 1; ret bias correction Climatological prior

Testing emissivity retrieval (Greenland) Snow emissivity spectra from RTTOV+MODIS+ASTER databases explicitly added to emissivity fit patterns Also tested adding patterns for other materials to data-base Doing so has little effect (patterns already in original approach) However retrieving without bias correction leads to much smaller residuals

ASTER Snow/ice models

Testing emissivity retrieval (Greenland) IR fixed bias correctionMWIR; ret bias correction Emis a priori = 1

Obs-simulation: Mean difference: SZA dependence MW channels (PWLR+TELSEM) IASI (PWLR+TELSEM) MW channels (IASI OEM +TELSEM) -> IASI/PWLR Daytime bias affects MW

Obs-simulation day vs night, MW vs IR window PWLR Day Night Day Night IR OEM MWIR OEM

Obs-simulation day vs night, MW vs IR window Day Night Day Night MWIR OEM MWIR+Emis MWIR+Emis-Ncor.

Retrieved emissivity day vs night 24 GHz Day 12 micron Day 12 micron Night 24 GHz Night

Day/night bias: summary Obs-simulations based on PWLR+RTTOV give IASI bias in day-time over land with opposite sign cf night-time between IASI and MW. +ve bias for IASI could be due to error in analysis (mapped to IASI time), transferred to PWLR via training. This removed by fitting surface T. Fitted emissivity day/night reasonably consistent for IASI Diurnal variation in MW bias not fully understood. Depends on land type, - ve MW bias in daytime not fully spatially correlated with +ve bias at night (though is in some places). Retrieving emissivity mainly resolves these biases Differences in MW may be enough to explain reduction in effect when emissivity fitted

MWIR with emissivity cf standard OEM

MWIR with emissivity & cloud cf standard OEM

MWIR with emissivity cf standard OEM

MWIR with emissivity & cloud cf standard OEM

Comparison of retrievals (cloud-free scenes)

Summary table UT Temperature

Day/land 17 April 2013

Summary table BL humidity

Day land 17 April 2013

Conclusions so far Differences between (RAL) retrievals and (Eumetsat) ODV are generally very small, particularly compared to the estimated retrieval error Desert surfaces problematic in IR – much improved by fitting emissivity, but still some spectral features remain Emissivity retrieval seems well constrained, even if it can give values slightly over 1 in some cases (e.g. when MW used). Bias correction not appropriate over ice surface / cloud Fit precision improves when residual pattern scale factors fitted Fitting emissivity clearly beneficial for IR-only ozone and water vapour, and improves convergence in difficult scenes Cloud fitting also seems to improve lower tropospheric temperature even in scenes for which ODV currently provided (“cloud free”), but this degrades convergence in some scenes (more later). PWLR std.dev vs Analysis greater on 17 April (and benefit of OEM more obvious) – some effect of “training” on other days? Ozone too constrained by PWLR; Climatological prior allows information to be extracted but validation cf analysis not reliable (analysis errors)