June 20, 2005Workshop on Chemical data assimilation and data needs Data Assimilation Methods Experience from operational meteorological assimilation John C. Derber Environmental Modeling Center NCEP/NWS/NOAA

June 20, 2005Workshop on Chemical data assimilation and data needs Data Assimilation Context Data assimilation attempts to bring together all available information to make the most probable estimate of: –The atmospheric state –The initial conditions to a model which will produce the best forecast.

June 20, 2005Workshop on Chemical data assimilation and data needs Data Assimilation Context Information sources –Observations –Background (forecast – carries forward info) –Dynamics (e.g., balances between variables) –Physical constraints (e.g., q > 0) –Statistical relationships between variables –Climatology Analysis should be most probable state given the above information sources and associated probability density functions

June 20, 2005Workshop on Chemical data assimilation and data needs Data Assimilation Context Must build data assimilation system within context of : –Observing system –Data handling system –Forecast model –Verification system –Computational resources –Human resources –Available knowledge about observations and statistics

June 20, 2005Workshop on Chemical data assimilation and data needs NCEP Production Suite Repeated four times per day

June 20, 2005Workshop on Chemical data assimilation and data needs Basic Assumptions (often violated) Data (forecast and most observations) are unbiased –e.g. radiosonde and others commonly biased –All forecast models have significant biases. –Note satellite observations bias corrected. Observational errors normally distributed –e.g., moisture errors not be normally distributed because moisture cannot be > saturation. Background uncorrelated to observational errors –May be true if not using retrievals –Representativeness error likely correlated

June 20, 2005Workshop on Chemical data assimilation and data needs Data assimilation theory With previous assumptions, most likely solution can be shown to be equivalent to solving variational problem Variational problem closely related to Kalman filtering/smoothing and ensemble filtering Often details more important than basic technique

June 20, 2005Workshop on Chemical data assimilation and data needs Atmospheric analysis problem (theoretical) J = J b + J o + J c J = (x-x b ) T B x -1 (x-x b ) + (K(x)-O) T (E+F) -1 (K(x)-O) + J C J = Fit to background + Fit to observations + constraints x= Analysis x b = Background B x = Background error covariance K= Forward model (nonlinear) O= Observations E+F= R = Instrument error + Representativeness error J C = Constraint term

June 20, 2005Workshop on Chemical data assimilation and data needs Atmospheric analysis problem (Practical) Must make analysis problem better conditioned to allow for faster convergence –Current Hessian = B -1 + K t (E+F) -1 K + … where K is the linearization of K –Want something closer to identity matrix –Preconditioning using B matrix Inclusion of all nonlinearities in minimization not necessary –substantial computation for little change –external iteration Need to simplify definition of B matrix to make computation feasible. –B matrix series of operators

June 20, 2005Workshop on Chemical data assimilation and data needs Atmospheric analysis problem (Practical) Analysis variable Analysis variable (x) and background (x b ) –Current global operational length ~ 47,000,000 (5*nlev+2)*(Ttrunc-1)*(Ttrunc-2) +nchan*npred 3-D – T382L64 –Temperature spectral coefficients –2 wind fields of spectral coefficients –Ozone spectral coefficients –Cloud liquid water spectral coefficients 2-D – T382 –Surface pressure of spectral coefficients –Skin temperature of spectral coefficients Predictors for bias correction for satellite data –x b background values – 6 hr forecast

June 20, 2005Workshop on Chemical data assimilation and data needs Sample forecast error structure

June 20, 2005Workshop on Chemical data assimilation and data needs Sample forecast error structure

June 20, 2005Workshop on Chemical data assimilation and data needs Sample forecast error structure

June 20, 2005Workshop on Chemical data assimilation and data needs Necessary future background error development Current errors are not situation dependent GSI analysis system under development to include situation dependent errors Using recursive filters to define background errors How to define errors still area of active research

June 20, 2005Workshop on Chemical data assimilation and data needs

June 20, 2005Workshop on Chemical data assimilation and data needs

June 20, 2005Workshop on Chemical data assimilation and data needs Atmospheric analysis problem (Practical) Inclusion of K operator most important advance in meteorological DA –Allows analysis variable to be different than observations –Simplest – 3-D interpolation to obs. location –More complicated – includes radiative transfer calculation –Even more complicated – adds integration of forecast model (4D-Var)

June 20, 2005Workshop on Chemical data assimilation and data needs Forward radiative transfer models Use of satellite radiance observations requires development of appropriate radiative transfer model NCEP using JCSDA Community Radiative Transfer Model (CRTM). Currently includes sensitivity to temperature, moisture, ozone and aerosols (not completely tested) Ongoing work to include clouds and other atmospheric constituents

June 20, 2005Workshop on Chemical data assimilation and data needs Atmospheric analysis problem (Practical) Observations used Conventional data –Radiosondes –Pibal winds –Aircraft winds and temperatures –Ships –Buoys –Synoptic stations –Profilers Satellite data –NOAA-14,15,16,17 1b radiances –GOES-10,(12) 5x5 radiances –Scatterometers –SSM/I wind speed and precipitation –TRMM precipitation –AMWs –SBUV ozone profiles

June 20, 2005Workshop on Chemical data assimilation and data needs Atmospheric analysis problem (Practical) Observations used Over 113M observations received. Over 7M observations per day used. Data selection and quality control eliminate many observations Data selection applied because of: –redundancy in data –reduction in computational cost –eliminate observations which are not useful E and F assumed diagonal – probably not true (especially for F).

June 20, 2005Workshop on Chemical data assimilation and data needs More about observations Observations communicated to NCEP and other NWP centers (GTS +) and available in real time –proprietary observations Data should be in standard format (BUFR –WMO standard) Quality control – eliminate bad data and data which cannot be properly modeled –Consistency checks (aircraft track, hydrostatic, etc.) –platform specific checks –cross platform checks

June 20, 2005Workshop on Chemical data assimilation and data needs Atmospheric analysis problem (Practical) Constraint term Currently only constraint used is limitation on the relative humidity –moisture > 0. and RH < ~1.2 –Used as penalty term –Applied in inner iteration - nonlinear –Can make problem significantly less well conditioned May be able to produce similar result through careful design of the analysis variable.

June 20, 2005Workshop on Chemical data assimilation and data needs Atmospheric analysis problem (Practical) Solution technique Solution must be made in real time (<20 minutes for global, less for limited area models at NCEP) At solution = 0 Use simple conjugate gradient scheme to find minimum Note one must have adjoint of observation operator to variationally solve problem. Usually not available with new observations.

June 20, 2005Workshop on Chemical data assimilation and data needs

June 20, 2005Workshop on Chemical data assimilation and data needs

June 20, 2005Workshop on Chemical data assimilation and data needs

June 20, 2005Workshop on Chemical data assimilation and data needs

June 20, 2005Workshop on Chemical data assimilation and data needs

June 20, 2005Workshop on Chemical data assimilation and data needs

June 20, 2005Workshop on Chemical data assimilation and data needs Potential problems for Chem. DA Sufficient sensitivity to Chem. in observing system Real-time observation availability and standardized formats and communication procedures Observation quality control Development of accurate forward models (and adjoints) and forecast models including Chem. Defining spatial and inter-variable correlations for Chem. variables Forecast model, forward model, and observational biases Developing appropriate validation and data monitoring capabilities Sufficient computational/human resources Collaboration

June 20, 2005Workshop on Chemical data assimilation and data needs Final comments Assimilation is the integration of all knowledge of the atmosphere (observations, physics, statistics) to estimate the state of the atmosphere Data assimilation systems are fairly good at large scales for the basic meteorological variables but other variables still in infancy. – Managing expectations Because of limited resources, operational systems must satisfy multiple purposes and any new developments must fit into operational infrastructure Must learn to think in terms of increments In data assimilation details are extremely important – you must do everything well!