DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What is Data Assimilation? A Tutorial Andrew S. Jones Lots of help also from: Steven Fletcher, Laura Fowler, Tarendra Lakhankar, Scott Longmore, Manajit Sengupta, Tom Vonder Haar, Dusanka Zupanski, and Milija Zupanski

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Data Assimilation Outline u Why Do Data Assimilation? u Who and What u Important Concepts u Definitions u Brief History u Common System Issues / Challenges

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Purpose of Data Assimilation u Why do data assimilation?

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Purpose of Data Assimilation u Why do data assimilation? (Answer: Common Sense)

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Purpose of Data Assimilation u Why do data assimilation? (Answer: Common Sense) MYTH: “It’s just an engineering tool”

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Purpose of Data Assimilation u Why do data assimilation? (Answer: Common Sense) MYTH: “It’s just an engineering tool” If Truth matters, “It’s our most important science tool”

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Purpose of Data Assimilation u Why do data assimilation? 1.I want better model initial conditions for better model forecasts

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Purpose of Data Assimilation u Why do data assimilation? 1.I want better model initial conditions for better model forecasts 2.I want better calibration and validation (cal/val)

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Purpose of Data Assimilation u Why do data assimilation? 1.I want better model initial conditions for better model forecasts 2.I want better calibration and validation (cal/val) 3.I want better acquisition guidance

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Purpose of Data Assimilation u Why do data assimilation? 1.I want better model initial conditions for better model forecasts 2.I want better calibration and validation (cal/val) 3.I want better acquisition guidance 4.I want better scientific understanding of  Model errors (and their probability distributions)  Data errors (and their probability distributions)  Combined Model/Data correlations  DA methodologies (minimization, computational optimizations, representation methods, various method approximations)  Physical process interactions

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, u Why do data assimilation? 1.I want better model initial conditions for better model forecasts 2.I want better calibration and validation (cal/val) 3.I want better acquisition guidance 4.I want better scientific understanding of  Model errors (and their probability distributions)  Data errors (and their probability distributions)  Combined Model/Data correlations  DA methodologies (minimization, computational optimizations, representation methods, various method approximations)  Physical process interactions (i.e., sensitivities and feedbacks) Leads toward better future models The Purpose of Data Assimilation

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, u Why do data assimilation? 1.I want better model initial conditions for better model forecasts 2.I want better calibration and validation (cal/val) 3.I want better acquisition guidance 4.I want better scientific understanding of  Model errors (and their probability distributions)  Data errors (and their probability distributions)  Combined Model/Data correlations  DA methodologies (minimization, computational optimizations, representation methods, various method approximations)  Physical process interactions (i.e., sensitivities and feedbacks) Leads toward better future models VIRTUOUS CYCLE The Purpose of Data Assimilation

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Data Assimilation Community u Who is involved in data assimilation? t NWP Data Assimilation Experts t NWP Modelers t Application and Observation Specialists t Cloud Physicists / PBL Experts / NWP Parameterization Specialists t Physical Scientists (Physical Algorithm Specialists) t Radiative Transfer Specialists t Applied Mathematicians / Control Theory Experts t Computer Scientists t Science Program Management (NWP and Science Disciplines) t Forecasters t Users and Customers

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Data Assimilation Community u What skills are needed by each involved group? t NWP Data Assimilation Experts (DA system methodology) t NWP Modelers (Model + Physics + DA system) t Application and Observation Specialists (Instrument capabilities) t Physical Scientists (Instrument + Physics + DA system) t Radiative Transfer Specialists (Instrument config. specifications) t Applied Mathematicians (Control theory methodology) t Computer Scientists (DA system + OPS time requirements) t Science Program Management (Everything + $$ + Good People) t Forecasters (Everything + OPS time reqs. + Easy/fast access) t Users and Customers (Could be a wide variety of responses) e.g., NWS / Army / USAF / Navy / NASA / NSF / DOE / ECMWF

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Data Assimilation Community u Are you part of this community?

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Data Assimilation Community u Are you part of this community? t Yes, you just may not know it yet.

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Data Assimilation Community u Are you part of this community? t Yes, you just may not know it yet. u Who knows all about data assimilation?

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Data Assimilation Community u Are you part of this community? t Yes, you just may not know it yet. u Who knows all about data assimilation? t No one knows it all, it takes many experts

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Data Assimilation Community u Are you part of this community? t Yes, you just may not know it yet. u Who knows all about data assimilation? t No one knows it all, it takes many experts u How large are these systems?

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Data Assimilation Community u Are you part of this community? t Yes, you just may not know it yet. u Who knows all about data assimilation? t No one knows it all, it takes many experts u How large are these systems? t Typically, the DA systems are “medium”-sized projects using software industry standards  Medium = multi-year coding effort by several individuals (e.g., RAMDAS is ~230K lines of code, ~3500 pages of code)  Satellite “processing systems” tend to be larger still t Our CIRA Mesoscale 4DVAR system was built over ~7-8 years with heritage from the ETA 4DVAR system

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Building Blocks of Data Assimilation NWP Model Observations NWP Adjoint Minimization Observation Model Adjoint Control Variables are the initial model state variables that are optimized using the new data information as a guide They can also include boundary condition information, model parameters for “tuning”, etc. Observation Model

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Are We Minimizing? Minimize discrepancy between model and observation data over time The Cost Function, J, is the link between the observational data and the model variables Observations are either assumed unbiased, or are “debiased” by some adjustment method

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Bayes Theorem Maximum Conditional Probability is given by: P (x | y) ~ P (y | x) P (x) Assuming Gaussian distributions… P (y | x) ~ exp {-1/2 [y – H (x)] T R -1 [y – H (x)]} P (x) ~ exp {-1/2 [x –x b ] T B -1 [x – x b ]} e.g., 3DVAR Lorenc (1986)

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Do We Trust for “Truth”? Minimize discrepancy between model and observation data over time Model Background or Observations?

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Do We Trust for “Truth”? Minimize discrepancy between model and observation data over time Model Background or Observations? Trust = Weightings Just like your financial credit score!

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Who are the Candidates for “Truth”? Minimize discrepancy between model and observation data over time Candidate 1: Background Term “x 0 ” is the model state vector at the initial time t 0 this is also the “control variable”, the object of the minimization process “x b ” is the model background state vector “B” is the background error covariance of the forecast and model errors

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Candidate 2: Observational Term “y” is the observational vector, e.g., the satellite input data (typically radiances), salinity, sounding profiles “M 0,i (x 0 )” is the model state at the observation time “i” “h”is the observational operator, for example the “forward radiative transfer model” “R”is the observational error covariance matrix that specifies the instrumental noise and data representation errors (currently assumed to be diagonal…) Who are the Candidates for “Truth”? Minimize discrepancy between model and observation data over time

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Do We Trust for “Truth”? Minimize discrepancy between model and observation data over time Candidate 1: Background Term The default condition for the assimilation when data are not available or the available data have no significant sensitivity to the model state or the available data are inaccurate

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Model Error Impacts our “Trust” Minimize discrepancy between model and observation data over time Candidate 1: Background Term Model error issues are important Model error varies as a function of the model time Model error “grows” with time Therefore the background term should be trusted more at the initial stages of the model run and trusted less at the end of the model run

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, How to Adjust for Model Error? Minimize discrepancy between model and observation data over time Candidate 1: Background Term Add a model error term to the cost function so that the weight at that specific model step is appropriately weighted or Use other possible adjustments in the methodology, i.e., “make an assumption” about the model error impacts If model error adjustments or controls are used the DA system is said to be “weakly constrained”

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What About Model Error Errors? Minimize discrepancy between model and observation data over time Candidate 1: Background Term Model error adjustments to the weighting can be “wrong” u u In particular, most assume some type of linearity u u Non-linear physical processes may break these assumptions and be more complexly interrelated A data assimilation system with no model error control is said to be “strongly constrained” (perfect model assumption)

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What About other DA Errors? Overlooked Issues? Data debiasing relative to the DA system “reference”. It is not the “Truth”, however it is self-consistent DA Methodology Errors? Assumptions: Linearization, Gaussianity, Model errors Representation errors (space and time) Poorly known background error covariances Imperfect observational operators Overly aggressive data “quality control” Historical emphasis on dynamical impact vs. physical Synoptic vs. Mesoscale?

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, DA Theory is Still Maturing The Future: Lognormal DA (Fletcher and Zupanski, 2006, 2007) Gaussian systems typically force lognormal variables to become Gaussian introducing an avoidable data assimilation system bias Many important variables are lognormally distributed Gaussian data assimilation system variables are “Gaussian” Add DA Bias Here! Lognormal Variables Clouds Precipitation Water vapor Emissivities Many other hydrologic fields Mode  Mean

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Do We Trust for “Truth”? Minimize discrepancy between model and observation data over time Candidate 2: Observational Term The non-default condition for the assimilation when data are available and data are sensitive to the model state and data are precise (not necessarily “accurate”) and data are not thrown away by DA “quality control” methods

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What “Truth” Do We Have? Minimize discrepancy between model and observation data over time DATA MODEL CENTRIC TRUTH

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, DA Theory is Still Maturing A Brief History of DA 1. Hand Interpolation 2. Local polynomial interpolation schemes (e.g., Cressman) 3. Use of “first guess”, i.e., a background 4. Use of an “analysis cycle” to regenerate a new first guess 5. Empirical schemes, e.g., nudging 6. Least squares methods 1. Variational DA (VAR) 2. Sequential DA (KF) 3. Monte Carlo Approx. to Seq. DA (EnsKF)

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Variational Techniques Finds the maximum likelihood (if Gaussian, etc.) (actually it is a minimum variance method) Comes from setting the gradient of the cost function equal to zero Control variable is x a Major Flavors: 1DVAR (Z), 3DVAR (X,Y,Z), 4DVAR (X,Y,Z,T) Lorenc (1986) and others… Became the operational scheme in early 1990s to the present day

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Sequential Techniques B is no longer static, B => P f = forecast error covariance P a (t i ) is estimated at future times using the model K = “Kalman Gain” (in blue boxes) Extended KF, P a is found by linearizing the model about the nonlinear trajectory of the model between t i-1 and t i Kalman (1960) and many others… These techniques can evolve the forecast error covariance fields similar in concept to OI

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Sequential Techniques f is a particular forecast instance l is the reference state forecast P f is estimated at future times using the model K number model runs are required (Q: How to populate the seed perturbations?) Sampling allows for use of approximate solutions Eliminates the need to linearize the model (as in Extended KF) No tangent linear or adjoint models are needed Ensembles can be used in KF-based sequential DA systems Ensembles are used to estimate P f through Gaussian “sampling” theory

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Sequential Techniques Notes on EnsKF-based sequential DA systems EnsKFs are an approximation Underlying theory is the KF Assumes Gaussian distributions Many ensemble samples are required Can significantly improve P f Where does H fit in? Is it fully “resolved”? What about the “Filter” aspects? Future Directions u u Research using Hybrid EnsKF-Var techniques

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Sequential Techniques N E is the number of ensembles S is the state-space dimension Each ensemble is carefully selected to represent the degrees of freedom of the system Square-root filter is built-in to the algorithm assumptions Zupanski (2005): Maximum Likelihood Ensemble Filter (MLEF) Structure function version of Ensemble-based DA (Note: Does not use sampling theory, and is more similar to a variational DA scheme using principle component analysis (PCA)

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Where is “M” in all of this? 3DDA Techniques have no explicit model time tendency information, it is all done implicitly with cycling techniques, typically focusing only on the P f term 4DDA uses M explicitly via the model sensitivities, L, and model adjoints, L T, as a function of time Kalman Smoothers (e.g., also 4DEnsKS) would likewise also need to estimate L and L T No M used M used

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, DVAR Revisited (for an example see Poster NPOESS P1.16 by Jones et al.) L T is the adjoint which is integrated from t i to t 0 Adjoints are NOT the “model running in reverse”, but merely the model sensitivities being integrated in reverse order, thus all adjoints appear to function backwards. Think of it as accumulating the “impacts” back toward the initial control variables. Automatically propagates the P f within the cycle, however can not save the result for the next analysis cycle (memory of “B” info becomes lost in the next cycle) (Thepaut et al., 1993)

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Minimization Process TRUTH Jacobian of the Cost Function is used in the minimization procedure Minima is at  J/  x = 0 Issues: Is it a global minima? Are we converging rapid or slow? J x

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, obs 1 obs 2 x time Geographically distant observations can bring more information than close-by observations, if in a dynamically significant region grid-point Ensembles: Flow-dependent forecast error covariance and spread of information from observations t0t0 t1t1 t2t2 Isotropic correlations From M. Zupanski

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Preconditioning the Space “Preconditioners” transform the variable space so that fewer iterations are required while minimizing the cost function x ->  Result: faster convergence From M. Zupanski

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Incremental VAR Courtier et al. (1994) Most Common 4D framework in operational use Incremental form performs Linear minimization within a lower dimensional space (the inner loop minimization) Outer loop minimization is at the full model resolution (non-linear physics are added back in this stage) Benefits: Smoothes the cost function and assures better minimization behaviors Reduces the need for explicit preconditioning Issues: Extra linearizations occur It is an approximate form of VAR DA

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Types of DA Solution Spaces Model Space (x) Physical Space (y) Ensemble Sub-space e.g., Maximum Likelihood Ensemble Filter (MLEF) Types of Ensemble Kalman Filters Perturbed observations (or stochastic) Square root filters (i.e., analysis perturbations are obtained from the Square root of the Kalman Filter analysis covariance)

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, How are Data used in Time? Assimilation time window observations Observation model Cloud resolving model time forecast

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Assimilation time window observations time forecast A “Smoother” Uses All Data Available in the Assimilation Window (a “Simultaneous” Solution) Observation model Cloud resolving model

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Assimilation time window observations time forecast A “Filter” Sequentially Assimilates Data as it Becomes Available in each Cycle Observation model Cloud resolving model

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Assimilation time window observations time forecast Cycle Previous Information Observation model Cloud resolving model A “Filter” Sequentially Assimilates Data as it Becomes Available in each Cycle

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Assimilation time window observations time forecast Cycle Previous Information Observation model Cloud resolving model A “Filter” Sequentially Assimilates Data as it Becomes Available in each Cycle

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Cycle Physics “Barriers” What Can Overcome the Barrier? 1. 1.Linear Physics Processes and 2. 2.Propagated Forecast Error Covariances time forecast Observation model Cloud resolving model A “Filter” Sequentially Assimilates Data as it Becomes Available in each Cycle

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Data Assimilation Conclusions u Broad, Dynamic, Evolving, Foundational Science Field! u Flexible unified frameworks, standards, and funding will improve training and education u Continued need for advanced DA systems for research purposes (non-OPS) u Can share OPS framework components, e.g., JCSDA CRTM Data Assimilation Thanks! For more information… Great NWP DA Review Paper (By Mike Navon) ECMWF DA training materials JCSDA DA workshop

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Backup Slides Or: Why Observationalists and Modelers see things differently… No offense meant for either side

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Approach Should We Use? DATA MODEL CENTRIC TRUTH

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Approach Should We Use? DATA MODEL CENTRIC TRUTH

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, My Precio us… We Trust the Model! Data hurts us!, Yes… What Approach Should We Use? DATA MODEL CENTRIC TRUTH

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, MODEL CENTRIC FOCUS DATA MODEL CENTRIC FOCUS ON “B” Background Error Improvements are Needed “x b ”Associated background states and “Cycling” are more heavily emphasized DA method selection tends toward sequential estimators, “filters”, and improved representation of the forecast model error covariances E.g., Ensemble Kalman Filters, other Ensemble Filter systems

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Approach Should We Use? DATA MODEL CENTRIC TRUTH This is not to say that all model-centric improvements are bad…

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Approach Should We Use? DATA MODEL CENTRIC TRUTH

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, My Precious… We Trust the Data! Models unfair and hurts us!, Yes… What Approach Should We Use? DATA MODEL CENTRIC TRUTH

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, DATA CENTRIC FOCUS DATA DATA CENTRIC FOCUS ON “h” Observational Operator Improvements are Needed “M 0,i (x 0 )” Model state capabilities and independent experimental validation is more heavily emphasized DA method selection tends toward “smoothers” (less focus on model cycling), more emphasis on data quantity and improvements in the data operator and understanding of data representation errors e.g., 4DVAR systems

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, DUAL-CENTRIC FOCUS Best of both worlds? Solution: Ensemble based forecast covariance estimates combined with 4DVAR smoother for research and 4DVAR filter for operations? Several frameworks to combine the two approaches are in various stages of development now…

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What Have We Learned? DATA MODEL CENTRIC TRUTH Your Research Objective is CRITICAL to making the right choices… Operational choices may supercede good research objectives Computational speed is always critical for operational purposes Accuracy is critical for research purposes

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What About Model Error Errors? A Strongly Constrained System? Can Data Over Constrain? “I just can’t run like I used to.” Model “Little Data People”

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What About Model Error Errors? A Strongly Constrained System? Can Data Over Constrain? “We’ll… no one’s perfect.” DA expert

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Optimal Interpolation (OI) OI merely means finding the “optimal” Weights, W Eliassen (1954), Bengtsson et al. (1981), Gandin (1963) Became the operational scheme in early 1980s and early 1990s A better name would have been “statistical interpolation”

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, What is a Hessian? A Rank-2 Square Matrix Containing the Partial Derivatives of the Jacobian G(f) ij (x) = D i D j f(x) The Hessian is used in some minimization methods, e.g., quasi-Newton…

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, The Role of the Adjoint, etc. Adjoints are used in the cost function minimization procedure But first… Tangent Linear Models are used to approximate the non-linear model behaviors L x’ = [M(x 1 ) – M(x 2 )] /  L is the linear operator of the perturbation model M is the non-linear forward model  is the perturbation scaling-factor x 2 = x 1 +  x’

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Useful Properties of the Adjoint  L T is the adjoint operator of the perturbation model Typically the adjoint and the tangent linear operator can be automatically created using automated compilers y =  (x 1, …, x n, y)  *x i =  *x i +  *y  /  x i  *y =  *y  /  ywhere  *x i and  *y are the “adjoint” variables

DoD CG/AR at Colorado State University AMS Data Assimilation Education Forum January 21, Useful Properties of the Adjoint  L T is the adjoint operator of the perturbation model Typically the adjoint and the tangent linear operator can be automatically created using automated compilers Of course, automated methods fail for complex variable types (See Jones et al., 2004) E.g., how can the compiler know when the variable is complex, when codes are decomposed into real and imaginary parts as common practice? (It can’t.)