1. Radiance Data Assimilation in WRF-Var
Please see other related detailed tutorial presentations available at
Satellite Data

2. Radiance codes are NOT included in the current V3.0.1.1 release.
Radiance codes are expected to be released in the next coming release (April 2009).
da/da_radiance] % ls
adm_ehv2pem.inc da_predictor_rttov.inc da_sort_rad.inc
da_allocate_rad_iv.inc da_print_stats_rad.inc da_transform_xtoy_crtm.inc
da_ao_stats_rad.inc da_qc_airs.inc da_transform_xtoy_crtm_adj.inc
da_biascorr.inc da_qc_amsua.inc da_transform_xtoy_rttov.inc
da_biasprep.inc da_qc_amsub.inc da_transform_xtoy_rttov_adj.inc
da_calculate_grady_rad.inc da_qc_crtm.inc da_write_biasprep.inc
da_cld_eff_radius.inc da_qc_hirs.inc da_write_filtered_rad.inc
da_cloud_detect_airs.inc da_qc_mhs.inc da_write_iv_rad_ascii.inc
da_cloud_sim_airs.inc da_qc_rad.inc da_write_oa_rad_ascii.inc
da_crtm.f da_qc_ssmis.inc ehv2pem.inc
da_crtm_ad.inc da_radiance.f emiss_ssmi.inc
da_crtm_direct.inc da_radiance1.f gsi_constants.f90
da_crtm_init.inc da_radiance_init.inc gsi_emiss.inc
da_crtm_k.inc da_read_biascoef.inc gsi_kinds.f90
da_crtm_sensor_descriptor.inc da_read_filtered_rad.inc gsi_thinning.f90
da_crtm_tl.inc da_read_kma1dvar.inc iceem_amsu.inc
da_detsurtyp.inc da_read_obs_bufrairs.inc init_constants_derived.inc
da_get_innov_vector_crtm.inc da_read_obs_bufrssmis.inc inria_n2qn1.inc
da_get_innov_vector_crtmk.inc da_read_obs_bufrtovs.inc landem.inc
da_get_innov_vector_radiance.inc da_read_pseudo_rad.inc module_radiance.f90
da_get_innov_vector_rttov.inc da_residual_rad.inc ossmem.inc
da_get_julian_time.inc da_rttov.f seaem.inc
da_get_time_slots.inc da_rttov_ad.inc siem_ats.inc
da_initialize_rad_iv.inc da_rttov_direct.inc siem_bts.inc
da_jo_and_grady_rad.inc da_rttov_init.inc siem_interpolate.inc
da_oi_stats_rad.inc da_rttov_tl.inc snwem_amsu.inc
da_predictor_crtm.inc da_setup_radiance_structures.inc

3. Basic Concepts: Radiative Transfer+ +
Surface
Cloud/rain
TOA radiance
at frequency 
Planck function
Atmospheric
absorption
Emission/reflection
Diffusion/scattering
Temperature information derived from well-mixed absorbents (CO2, …)
Channels sensitive to Humidity, Ozone, …
Surface channels: “window” parts of spectrum
AMSUA

4. Basic Concepts: Radiative Transfer+ +
Surface
Cloud/rain
TOA radiance
at frequency 
Planck function
Atmospheric
absorption
Emission/reflection
Diffusion/scattering
CRTM, RTTOV, …
T, Q, O3, …
Forward model
L() RT
Inversion
Regression, neural network, 1D-Var

5. Practical issues: Quality Control
Specific QC for each sensor AMSU-A, AMSU-B, MHS, SSMIS, AIRS
Pixel-level QC
Reject limb observations
Reject pixels over land and sea-ice
Cloud/Precipitation detection
Synergy with imager (AIRS/VIS-NIR)
Channel-level QC
Gross check (innovations <15 K)
First-guess check (innovations < 3o).
Observation error tuning
Error factor tuned from objective method (Desrozier and Ivanov, 2001)
AMSU-B
89GHz-150GHz
Tb < 3K
Katrina Location (2005/08/26/06Z)
CLWP(mm)
from Guess
<0.2mm

6. Practical issues: Bias Correction
Predictors:
Offset
mb thickness
200-50mb thickness
Surface skin temperature
Total column water vapor
Scan
Parameters
Modeling of errors in satellite radiances:
Air Mass Bias + Scan Bias
Scan Bias = d(limb) - d(nadir)
d(.) is departure (omb or oma)
This is relative bias between limb and nadir

7. Practical issues: Variational Bias Correction
Predictors:
Offset
mb thickness
200-50mb thickness
Surface skin temperature
Total column water vapor
Scan
Parameters
Modeling of errors in satellite radiances:
Bias parameters can be estimated within the variational assimilation, jointly with the atmospheric model state (Derber and Wu 1998) (Dee 2005) (Auligné et al. 2007)
Inclusion of the bias parameters in the control vector : xT  [x, ]T
Jb: background term for x
Jo: corrected observation term
«Optimal » bias correction considering all available information
J: background term for 

8. Practical issues: Thinning
No Thinning
120km Thinning Mesh

9. Data Ingest (sources, instruments, Tb, Ta)
Radiative transfer model
Channel selection
Bias correction
Off-line bias correction statistics
Variational bias correction
Diagnostics and monitoring

10. Data Ingest NCEP global BUFR radiance data HIRS from NOAA16, 17, 18
(Total: 14 sensors from 6 satellites)
HIRS from NOAA16, 17, 18
AMSU-A from NOAA15, 16, 18, EOS-Aqua, METOP-2
AMSU-B from NOAA15, 16, 17
MHS from NOAA18, METOP-2
AIRS from EOS-Aqua
NCAR processed AMSU-A, AMSU-B, MHS BUFR files.
NRL/AFWA/NESDIS produced DMSP-16 SSMI/S BUFR radiance data.

11. Data Processing
Raw orbit-by-orbit level-1B radiance in binary format (including coefficients for radiance to temperature conversion) from NESDIS
[ --> antenna temperature ]
--> brightness temperature
Concatenate orbit-by-orbit files within desired time window into one BUFR file
Remove duplicate data

13. ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.${yyyymmddhh}
NCEP BUFR
ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.${yyyymmddhh}
NCEP naming convention
WRF-Var naming convention
gdas1.t00z.1bamua.tm00.bufr_d
gdas1.t00z.1bamub.tm00.bufr_d
gdas1.t00z.1bhrs3.tm00.bufr_d
gdas1.t00z.1bhrs4.tm00.bufr_d
gdas1.t00z.1bmhs.tm00.bufr_d
gdas1.t00z.airsev.tm00.bufr_d
gdas1.t00z.airs.tm00.bufr_d
amsua.bufr
amsub.bufr
hirs3.bufr
hirs4.bufr
mhs.bufr
airs.bufr
ssmis.bufr
Direct input to WRF-Var, no pre-processing required.
Quality control, thinning, time and domain check, bias correction are done inside WRF-Var
Namelist switches to decide if reading the data or not
Use_amsuaobs
Use_amsubobs
Use_hirs3obs
Use_hirs4obs
Use_mhsobs
Use_airsobs
Use_eos_amsuaobs
Use_ssmisobs

14. Radiative Transfer Model
For all the satellite sensors supported,
given an atmospheric profile of temperature, water vapour
and optionally ozone and carbon dioxide
together with satellite zenith angle and surface temperature, pressure and optionally surface emissivity,
RTTOV will compute the top of atmosphere radiances
in each of the channels of the sensor being simulated.
From

15. Radiative Transfer Model
CRTM (Community Radiative Transfer Model)
JCSDA (Joint Center for Satellite Data Assimilation)
ftp://ftp.emc.ncep.noaa.gov/jcsda/CRTM/
Latest released version: CRTM REL-1.2_beta, September 2008
Version used in WRF-Var: CRTM REL-1.1
Documentation still under development
RTTOV (Radiative Transfer for TOVS)
EUMETSAT (European Organisation for the Exploitation of Meteorological Satellites)
Latest released version: RTTOV_9_2, July 2008
Version used in WRF-Var: RTTOV_8_7 (with a small bug fix)
in WRF-Var applications, CRTM and RTTOV produce similar results while RTTOV is much faster than CRTM

16. Channel selection Window Channels (1~3,15) not used
Channels 10~14 above
model top removed
Ch4: 700mb
Ch5: 500~700mb
Ch6: 400~500mb
Ch7: 200~300mb
Sounding Channels
(5~9) sensitive to
Temperature.
Ch8: 200mb
Ch9: 100mb
Ch10: 50mb
Ch11: 30mb

17. Channel selection metop-2-mhs.info
WRFDA/var/run/radiance_info>ls -l
total 160
-rw-r--r hclin users Aug 22 17:01 dmsp-16-ssmis.info
-rw-r--r hclin users Aug 22 17:01 eos-2-airs.info
-rw-r--r hclin users Aug 22 17:01 eos-2-amsua.info
-rw-r--r hclin users Aug 22 17:01 metop-2-amsua.info
-rw-r--r hclin users Aug 22 17:01 metop-2-mhs.info
-rw-r--r hclin users Aug 22 17:01 noaa-15-amsua.info
-rw-r--r hclin users Aug 22 17:01 noaa-15-amsub.info
-rw-r--r hclin users Aug 22 17:01 noaa-15-hirs.info
-rw-r--r hclin users Aug 22 17:01 noaa-16-amsua.info
-rw-r--r hclin users Aug 22 17:01 noaa-16-amsub.info
-rw-r--r hclin users Aug 22 17:01 noaa-16-hirs.info
-rw-r--r hclin users Aug 22 17:01 noaa-17-amsub.info
-rw-r--r hclin users Aug 22 17:01 noaa-17-hirs.info
-rw-r--r hclin users Aug 22 17:01 noaa-18-amsua.info
-rw-r--r hclin users Aug 22 17:01 noaa-18-hirs.info
-rw-r--r hclin users Aug 22 17:01 noaa-18-mhs.info
metop-2-mhs.info
sensor channel IR/MW use idum varch polarisation(0:vertical;1:horizontal)
E E+00
E E+00
E E+01
E E+01
E E+00

18. Setup and run - with radiances
To run WRF-Var, first create a working directory,
for example, WRFDA/var/test, then follow the steps below:
cd WRFDA/var/test (go to the working directory)
ln -sf WRFDA/run/LANDUSE.TBL ./LANDUSE.TBL
ln -sf $DAT_DIR/rc/ /wrfinput_d01 ./fg (link first guess file as fg)
ln -sf WRFDA/var/obsproc/obs_gts_ _00:00:00.3DVAR ./ob.ascii (link OBSPROC processed
observation file as ob.ascii)
ln -sf $DAT_DIR/be/be.dat ./be.dat (link background error statistics as be.dat)
ln -sf WRFDA/var/da/da_wrfvar.exe ./da_wrfvar.exe (link executable)
ln -sf $DAT_DIR/ /gdas1.t00z.1bamua.tm00.bufr_d ./amsua.bufr
ln -sf $RADIANCE_INFO_DIR ./radiance_info
ln -sf $BIASCORR_DIR ./biascorr
(CRTM only) ln -sf $CRTM_COEFFS_DIR ./crtm_coeffs
(RTTOV only) ln -sf $RTTOV_COEFFS_DIR/rtcoef*.dat ./rtcoef*.dat
(Variational Bias Correction only) vi VARBC.in
vi namelist.input (&wrfvar4, &wrfvar14, &wrfvar12, &wrfvar22)
da_wrfvar.exe >&! wrfda.log

19. From RTTOV_8_7 Users Guide
sensor_id
Instrument triplets platform_id
satellite_id
sensor_id
platform_id
satellite_id

20. namelist.input RTMINIT_NSENSOR = 12
RTMINIT_PLATFORM = 1, 1, 1, 9,10, 1, 1, 1, 1,10, 9, 2
RTMINIT_SATID = 15,16,18, 2, 2,15,16,17,18, 2, 2,16
RTMINIT_SENSOR = 3, 3, 3, 3, 3, 4, 4, 4,15,15,11,10
NOAA-15-AMSUA
NOAA-16-AMSUA
NOAA-18-AMSUA
EOS-2-AMSUA
METOP-2-AMSUA
NOAA-15-AMSUB
NOAA-16-AMSUB
NOAA-17-AMSUB
NOAA-18-MHS
METOP-2-MHS
EOS-2-AIRS
DMSP-16-SSMIS

22. RAD_MONITORING (30)
Integer array with dimension RTMINIT_NSENSER, where 0 for assimilating mode, 1 for monitoring mode (only calculate innovation).
THINNING
Logical, TRUE will perform thinning
THINNING_MESH (30)
Real array with dimension RTMINIT_NSENSOR, values indicate thinning mesh (in KM) for different sensors.
READ_BIASCOEF
Logical, control if reading bias correction coefficient files (Harris and Kelly scheme), always set to TRUE.
BIASCORR
Logical, control if perform bias correction (Harris and Kelly scheme).
BIASPREP
Logical, control if perform bias correction preparation (Harris and Kelly scheme).
QC_RAD
Logical, control if perform quality control, always set to TRUE.
WRITE_IV_RAD_ASCII
Logical, control if output Observation minus Background files, which are ASCII format and separated by sensors and processors.
WRITE_OA_RAD_ASCII
Logical, control if output Observation minus Analysis files (including also O minus B), which are ASCII format and separated by sensors and processors.
USE_ERROR_FACTOR_RAD
Logical, control if use a radiance error tuning factor file “radiance_error.factor”, which is created with empirical values or generated using variational tunning method (Desroziers and Ivannov, 1997)
ONLY_SEA_RAD
Logical, control if only assimilating radiance over water.
TIME_WINDOW_MIN
String, e.g., " _03:00: ", start time of assimilation time window
TIME_WINDOW_MAX
String, e.g., " _09:00: ", end time of assimilation time window

23. RTTOV coefficients rttov_coefs>ls -l total 7488
-rw-r--r hclin users Sep rtcoef_dmsp_10_ssmi.dat
-rw-r--r hclin users Sep rtcoef_dmsp_11_ssmi.dat
-rw-r--r hclin users Sep rtcoef_dmsp_12_ssmi.dat
-rw-r--r hclin users Sep rtcoef_dmsp_13_ssmi.dat
-rw-r--r hclin users Sep rtcoef_dmsp_14_ssmi.dat
-rw-r--r hclin users Sep rtcoef_dmsp_15_ssmi.dat
-rw-r--r hclin users Sep rtcoef_dmsp_16_ssmis.dat
-rw-r--r hclin users Sep rtcoef_dmsp_8_ssmi.dat
-rw-r--r hclin users Sep rtcoef_dmsp_9_ssmi.dat
-rw-r--r hclin users Oct rtcoef_eos_2_amsua.dat
-rw-r--r hclin users Oct rtcoef_metop_2_amsua.dat
-rw-r--r hclin users Oct rtcoef_metop_2_mhs.dat
-rw-r--r hclin users Oct rtcoef_noaa_15_amsua.dat
-rw-r--r hclin users Oct rtcoef_noaa_15_amsub.dat
-rw-r--r hclin users Oct rtcoef_noaa_16_amsua.dat
-rw-r--r hclin users Oct rtcoef_noaa_16_amsub.dat
-rw-r--r hclin users Oct rtcoef_noaa_17_amsua.dat
-rw-r--r hclin users Oct rtcoef_noaa_17_amsub.dat
-rw-r--r hclin users Oct rtcoef_noaa_18_amsua.dat
-rw-r--r hclin users Oct rtcoef_noaa_18_mhs.dat

24. CRTM_02-29-08/CRTM_Coeffs>ls -l
drwxr-xr-x 5 wrfhelp users 512 Mar AerosolCoeff
drwxr-xr-x 5 wrfhelp users 512 Mar CloudCoeff
drwxr-xr-x 5 wrfhelp users 512 Mar EmisCoeff
drwxr-xr-x 4 wrfhelp users 512 Mar SpcCoeff
drwxr-xr-x 5 wrfhelp users 512 Mar TauCoeff
CRTM coefficients
CRTM_ /CRTM_Coeffs/SpcCoeff>ls -l
drwxr-xr-x 5 wrfhelp users 512 Dec Infrared
drwxr-xr-x 5 wrfhelp users 512 Dec Microwave
Link selected files from various source directories into single directory then link it as crtm_coeffs in the WRF-Var working directory
CRTM_ /CRTM_Coeffs/SpcCoeff/Microwave>ls -l
drwxr-xr-x 5 wrfhelp users 512 Dec AAPP_AC
drwxr-xr-x 5 wrfhelp users 512 Dec NESDIS_AC
drwxr-xr-x 5 wrfhelp users 512 Feb No_AC
CRTM_ /CRTM_Coeffs/SpcCoeff/Microwave/No_AC>ls -l
drwxr-xr-x 2 wrfhelp users Dec Big_Endian
drwxr-xr-x 2 wrfhelp users Dec Little_Endian
drwxr-xr-x 2 wrfhelp users Feb netCDF
CRTM_ /CRTM_Coeffs/SpcCoeff/Microwave/No_AC/Big_Endian>ls -l
total 76
-rw-r--r wrfhelp users Dec amsua_metop-a.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsua_metop-b.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsua_metop-c.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsua_n15.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsua_n16.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsua_n17.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsua_n18.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsua_n19.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsub_n15.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsub_n16.SpcCoeff.bin
-rw-r--r wrfhelp users Dec amsub_n17.SpcCoeff.bin

25. Bias Correction Satellite radiance is generally considered biased
with respect to a reference (e.g., background or
analysis field in NWP assimilation) due to system
error of observation itself, reference field and RTM.

26. Off-line bias correction statistics (Harris and Kelly, 2001)
Step 1: generate 'biasprep' files by running WRF-Var
To generate “biasprep” files separated by sensors and processors which contains necessary information (obs, background, o minus b, predictors etc.) for used by the off-line statistics.
For statistical significance, month-long training dataset is preferred.
Running WRF-Var with radiance data to be used and appropriate background field (e.g., produced from GFS analysis using WPS/REAL, or from a WRF 6h forecast in an existing assimilation experiment).
Note that the effect of bias correction can depend to some extent upon the reference field used for statistics.
BIASCORR=false
BIASPREP=true # control to generate biasprep* files (biasprep_noaa-15-amsua.*, biasprep_noaa-16-amsua.)
NTMAX=0 # no minimization needed since biasprep run just need to calculate innovation.
biasprep* files will be used for the off-line statistics described in step 2.
Step 2: off-line statistics
biascorr 139 > ls -l
total 1280
-rw-r--r hclin ncar Jul 03 12:29 dmsp-16-ssmis.bcor
-rw-r--r hclin ncar Jul 03 12:29 metop-2-amsua.bcor
-rw-r--r hclin ncar Jul 03 12:29 metop-2-mhs.bcor
-rw-r--r hclin ncar Jul 03 12:29 noaa-15-amsua.bcor
-rw-r--r hclin ncar Jul 03 12:29 noaa-15-amsub.bcor
-rw-r--r hclin ncar Jul 03 12:29 noaa-16-amsua.bcor
-rw-r--r hclin ncar Jul 03 12:29 noaa-16-amsub.bcor
-rw-r--r hclin ncar Jul 03 12:29 noaa-17-amsub.bcor
-rw-r--r hclin ncar Jul 03 12:29 noaa-18-amsua.bcor
-rw-r--r hclin ncar Jul 03 12:29 noaa-18-mhs.bcor

27. error standard deviation used as observation errors in WRF-Var
biascorr 142 > more noaa-16-amsua.bcor
RELATIVE SCAN BIASES
error standard deviation used as observation errors in WRF-Var
Coefficients for air-mass bias correction
column 1: mb thickness
column 2: mb thickness
column 3: Surface skin temperature
column 4: Total column water vapor
column 5: offset

28. Variational Bias Correction (VarBC)
VARBC.in file is an ASCII file that controls all of what is going into the VarBC.
Sample VARBC.in
VARBC version Number of instruments: 2
Platform_id Sat_id Sensor_id Nchanl Npredmax
-----> Bias predictor statistics: Mean & Std & Nbgerr
-----> Chanl_id Chanl_nb Pred_use(-1/0/1) Param

29. Sample VARBC.out (VARBC.in)
VARBC version Number of instruments: 4
Platform_id Sat_id Sensor_id Nchanl Npredmax
-----> Bias predictor statistics: Mean & Std & Nbgerr
-----> Chanl_id Chanl_nb Pred_use(-1/0/1) Param

30. Evolution of VarBC parameters

31. radiance monitoring
Generally monitoring stage is necessary before new instruments enter into operational assimilation system
Two aspects
Assimilating/Monitoring mode switch on/off ability for different satellite instruments in WRF-Var for real time operation
Statistics/Visualization tool to monitor and assess observation quality

32. radiance monitoring - continue
namelist parameter: rad_monitoring
e.g., 10 instruments ingested into WRF-Var
rad_monitoring=0,0,0,0,0,1,1,1,1,1
0: assimilating mode, 5 sensors
calculate innovation (O minus B) and enter into minimization
1: monitoring mode, 5 sensors
calculate innovation but NOT enter into minimization

33. radiance monitoring - continue
Radiance Post-Processing/Visualization
Output innovation and auxiliary information into ASCII files separated for CPUs
Convert ASCII files to one NETCDF file for each sensor (independent Fortran90 program), easy to manipulate with ncdump, nco, ncview tools
NCL script to plot various graphics
Channel TB, Histogram, scatter plot, time series etc.
Can be included in script to routinely produce graphics after WRF-Var runs
Users can control (by simple script parameter setup) to plot over smaller domain, only over land or sea, QCed or no-QCed observations

34. ------ OMB before bias correction
OMB after bias correction