1. UNBIASED ESTIAMTION OF ANALYSIS AND FORECAST ERROR VARIANCES
Zoltan Toth1 & Malaquias Pena2
1 Global Systems Division
NOAA/OAR/ESRL
2 Environmental Modeling Center
NOAA/NWS
Acknowledgements:
Mozheng Wei, Yuanfu Xie, Isidora Jankov, Yuejian Zhu

2. Why analysis / forecast error variance matters?
OUTLINE / SUMMARY
Why analysis / forecast error variance matters?
Needed in data assimilation & ensemble forecasting
How can we estimate analysis error variance?
Parametric estimation method
Use perceived error variance
Link perceived error variance to true error variance
Utilize prior knowledge
Independent quality assessment
Knowledge & fact-based - no assumptions

3. MOTIVATION – WHY WE NEED ERROR ESTIMATES?
Data assimilation
Assessing quality of analysis
Background forecast error variance important input parameter
Model development
Evaluating / comparing quality of NWP forecasts
On scales models can resolve
Ensemble generation
Initial perturbation & analysis error variances must match

4. INTRODUCTION TO APPROACH
Objective
Unbiased estimate of analysis / forecast error variance
Definitions
Analysis / forecast error
Difference between truth & analysis/forecast on model grid
Perceived forecast error
Difference between analysis and forecast
Challenges
Truth not known – can error variance still be estimated?
Observations sparse & fraught with instrument / representativeness errors
Estimates with variational / ensemble-based DA methods
Qualitative only (tuning parameters, truncated space)

5. SCHEMATIC
Forecast
Perceived error
Analysis
True error
Truth ρ2 ρ1

6. PROPOSED METHOD Pena & Toth, 2014
Use perceived forecast errors at different lead times as measurements
Identify prior knowledge relating perceived & true error variances
True forecast errors
Initiate from true analysis errors (1)
Evolve according to basic & well known error growth dynamics (2)
Effect of observations via analysis step
De-correlates forecast errors from analysis errors (3)
Perform parametric estimation
Describe unknown variables (true errors) as a function of measured variables (perceived errors)
Seek additional relationships (prior knowledge) until # unknowns << # measured variables
Estimate unknowns (true error) by finding the values that provide best fit of parametric function to measured quantities (perceived error)

7. THE EQUATIONS (0) Basic Eq in functional analysis
Relationship between perceived (measured) & true forecast errors (unknown)
(1) Same analysis system used to initialize and verify forecasts
Equivalence
(2) Temporal evolution of forecast error (error growth)
(3) Effect of successive DA steps on analysis/forecast error correlation

8. PERCEIVED – TRUE ERROR RELATIONSHIP
Law of cosines
Decompose perceived forecast error into true forecast error & analysis error variance terms for each lead-time i
Di perceived error Fi forecast
A analysis T truth
Xi true forecast error variance x0 true analysis error variance
correlation between analysis & forecast error valid at same time
Set of Eqs like above for each lead-time i
2 unknowns (xi, ) for each Eq. plus x0
Need additional relationships ρ2 ρ1

9. ERROR GROWTH RELATIONSHIPS
Chaotic error growth – Lorenz 1963, 1982
Linear error growth – exponential curve
growth rate (α), initial analysis error (x0)
Nonlinear error growth – logistic curve
One additional parameter, error saturation level (s∞)
Model drift error –Toth & Pena 2007
Saturating exponential
b asymptotic difference between corresponding states on the attractors of nature and model
a magnitude of drift related error in initial condition
and
1/ “e-folding” time of errors from initial to saturation time
Simple general error growth curve
Add chaotic & model drift components

10. ANALYSIS / FORECAST ERROR CORRELATION
With no DA step, analysis & forecast errors correlate at 1.0
With one DA step, errors become de-correlated, 1 > ρ1 > 0
With multiple (i) DA steps,
Assuming effectiveness of observing & DA systems stationary in time
Note same analysis system used for both
Initialization & verification
Choice reduces # unknowns ρ2 ρ1

11. MINIMIZATION Combine error decomposition, growth, & correlation Eqs
Substitute variables to collapse number of unknowns
# unknowns << # equations
Solve for unknowns by minimizing
di Measurements
di^ Modeled quantity linked with unknowns in functional analysis
L(2) norm used
Nelder-Mead Simplex method, Lagarias et al. 1998
Weights for minimization
Standard Error of Mean (for measurements)
Expected deviation between simulated & measured perceived error
Due to sampling error (from finite size perceived error sample)
sd Standard Deviation
N Sample size

12. EXPERIMENTS Errors in simple (and perfect) model experiments
Real errors
Linear regime – Short lead times, NH
Nonlinear regime Extended lead times, NH
Model errors Tropics
Multicenter inter-comparison NH

13. PERFECT MODEL EXPERIMENTS WITH SMALL OBSERVATIONAL ERRORS LARGE
Observed / modeled perceived error
Good fit
Poor fit
Observed / modeled true error
Observed / modeled analysis / forecast error correlation

14. REAL WORLD – WITH MODEL ERROR
Tropical 10m U Wind
Center A
Analysis / forecast error
Analysis / forecast error correlation
Estimated sampling uncertainty
Fitting error

15. MULTI-CENTER INTERCOMPARISON
Analysis / forecast error estimate – NH 500 hPa height
6.2 m
5.4 m B
Center A C
3 .4 m
7 m D

16. Validity of approach and results hinge on and only on prior knowledge
DEPENDENCIES
Validity of approach and results hinge on and only on prior knowledge
Need to be looked at very carefully
No dependence on DA assumptions / estimates
Quality (error) assessment not based on tools generating product (analysis)
Independent quality assessment

17. Why analysis / forecast error variance matters?
OUTLINE / SUMMARY
Why analysis / forecast error variance matters?
Needed in data assimilation & ensemble forecasting
How can we estimate analysis error variance?
Functional analysis
Use perceived error variance
Link perceived error variance to true error variance
Utilize prior knowledge
Independent quality assessment
Knowledge & fact-based - no assumptions

19. BACKGROUND

20. PRIOR KNOWLEDGE
(1) True forecast errors initiate from true analysis errors
Trivial but not considered/exploited in prior studies
Valid/relevant only when forecasts are verified against analyses used to initialize them (ie, “verified against its own analysis”)
Practice frowned upon by some as “own” analysis not considered independent
Yet important in reducing number of unknowns
(2) True forecast error evolution follow chaotic error dynamics
Basic & well known facts yet not considered/exploited in error estimation
Very useful in reducing number of unknowns
Caveat in presence of model errors
(3) Effect of observations via analysis step de-correlates forecast errors from analysis errors
Trivial observation again
No observational data: Analysis error = forecast error (no impact from analysis)
Recognition that successive analysis steps have a power effect on correlations between errors in analyses & forecasts valid at same time
Major reduction in number of unknowns

21. THE EQUATIONS (1) Basic Eq in functional analysis
Relationship between perceived (measured) & true forecast errors (unknown)
(2) Same analysis system used to initialize and verify forecasts
Equivalence
(3) Temporal evolution of forecast error (error growth)
(4) Effect of successive DA steps on analysis/forecast error correlation

22. PERCEIVED – TRUE ERROR RELATIONSHIP
Decompose perceived forecast error into true forecast error & analysis error variance terms for each lead-time i
Di perceived error Fi forecast
A analysis T truth
Xi true forecast error variance x0 true analysis error variance
correlation between analysis & forecast error valid at same time
Set of Eqs like above for each lead-time i
2 unknowns (xi, ) for each Eq. plus x0
Need additional relationships

23. ERROR GROWTH RELATIONSHIPS
Chaotic error growth – Lorenz 1963, 1982
Linear error growth – exponential curve
growth rate (α), initial analysis error (x0)
Nonlinear error growth – logistic curve
One additional parameter, error saturation level (s∞)
Model drift error –Toth & Pena 2007
Saturating exponential
b asymptotic difference between corresponding states on the attractors of nature and model
a magnitude of drift related error in initial condition
and
1/ “e-folding” time of errors from initial to saturation time
Simple general error growth curve
Add chaotic & model drift components

24. ANALYSIS / FORECAST ERROR CORRELATION
With no DA step, analysis & forecast errors correlate at 1.0
With one DA step, errors become de-correlated, 1 > ρ1 > 0
With multiple (i) DA steps,
Assuming effectiveness of observing & DA systems stationary in time
Note same analysis system used for both
Initialization & verification
Choice reduces # unknowns

25. MINIMIZATION Combine error decomposition, growth, & correlation Eqs
Substitute variables to collapse number of unknowns
# unknowns << # equations
Solve for unknowns by minimizing
di Measurements
di^ Modeled quantity linked with unknowns in functional analysis
L(2) norm used
Nelder-Mead Simplex method, Lagarias et al. 1998
Weights for minimization
Standard Error of Mean (for measurements)
Expected deviation between simulated & measured perceived error
Due to sampling error (from finite size perceived error sample)
sd Standard Deviation
N Sample size

26. EXPERIMENTS Errors in simple (and perfect) model experiments
Real errors
Linear regime – Short lead times, NH
Nonlinear regime Extended lead times, NH
Model errors Tropics
Multicenter inter-comparison NH

27. PERFECT MODEL / NATURE – LORENZ 1963
=10
b =8/3
r =28
Runge-Kutta numerical scheme with a time step of 0.01

28. DATA ASSIMILATION 3-DVAR algorithm Cost function
8 time step cycle length (~3 hrs in atmosphere)
Diagonal R (observational Variance), R=2
B based on independent forecast errors (NMC method)
Empirically tuned variance
Cost function

29. PERFECT MODEL EXPERIMENTS
Create “nature”
Long integration of model
Produce “observations”
Add random noise with zero mean and X variance
Assimilate observations every 15 time steps w 3DVar
Initialize forecasts with analyses
40 DA-cycle unit long integrations

30. SMALL, MEDIUM, LARGE OBSERVATIONAL ERROR
Nonlinear saturation
Forecast error
True (continuous) Perceived (dashed)
Correlation between analysis & forecast errors

31. REAL WORLD – LINEAR ERROR GROWTH
NH 500 hPa Height
Center A
Analysis / forecast error
Estimated sampling uncertainty
Fitting error
Analysis / forecast error correlation

32. MULTI-CENTER INTERCOMPARISON
Estimated vs. actual sampling error in fitting
NH 500 hPa Height
Center A B C D

33. MULTI-CENTER INTERCOMPARISON
Correlation estimate – NH 500 hPa Height
Center A B C D

34. RELEVANCE OF RESULTS Quality assessment
Data assimilation
Specify background error variance
Short-range forecasts
Not important for longer range forecasts
Effect of analysis errors small?
Comparison of results from different centers / systems
Common reference (truth)
Ensemble initialization
Specify initial perturbation variance
Then use forecasts for setting background covariances in variational DA

35. How can we estimate analysis error variance?
OUTLINE / SUMMARY
What is shadowing?
Concept of Data Assimilation in Numerical Weather Prediction
State estimation with incomplete information
How can we estimate analysis error variance?
New approach – functional analysis linking perceived vs true errors