1. Steve Keighton National Weather Service Blacksburg, VA
Predictability, Ensemble Forecasts, and the use of Statistical Guidance in the Forecast Process
Steve Keighton
National Weather Service
Blacksburg, VA

2. Outline Acknowledgments
Chaos theory and predictability in the atmosphere
Numerical Weather Prediction (NWP) and use of “ensemble” forecast methods
Use of statistical guidance in the forecast process (Model Output Statistics) – if time
Acknowledgments
Josh Korotky – NWS Pittsburgh
Mark Antolik – NWS Meteorological Development Lab

3. “Prediction is very difficult, especially about the future”
“Prediction is very difficult, especially about the future” Niels Bohr

4. Prelude – What is Chaos and why is it important?
Chaos leads us from the laws of nature to their consequences
…shows us that simple systems can exhibit complex behavior…and vice versa
…demonstrates that unpredictable behavior can develop in a system governed by deterministic laws
As forecasters…chaos shows us the limits of predictability
…highlights the importance of probabilistic thinking
…shows us the value of expressing uncertainty in forecasts
…helps us understand why the future of forecasting will lean heavily on ensemble rather than deterministic approaches

5. Edward Lorenz (1917 – 2008)
Small errors in the initial state estimate of a nonlinear system can limit the prediction of later states of the system
Chaos occurs when error propagation, seen as a signal in time, grows to the same size or scale as the original signal...
“… one flap of a sea-gull’s wing may forever change the future course of the weather” (Lorenz, 1963)

6. Elements of Chaos
Dynamical system – future states caused by past states (determinism)
Nonlinearity – system output (response) isn't proportional to input (forcing)…
a small forcing can lead to a disproportionately large response and vise versa
a system's values at one time aren‘t proportional to the values at an earlier time
Non-periodic behavior – future states never repeat past states
Extreme sensitivity to initial conditions – small initial state uncertainties amplify…
a "prediction horizon” is inevitable
Even though the governing laws of a system are known, long-term predictions can be meaningless
Chaos occurs only in deterministic, nonlinear, dynamical systems
Chaos results from a deterministic process.
2. It happens only in nonlinear systems.
3. The motion or pattern for the most part looks disorganized and erratic, although sustained. In fact, it can
usually pass all statistical tests for randomness.
4. It happens in feedback systems—systems in which past events affect today's events, and today's events
affect the future.
5. It can result from relatively simple systems. With discrete time, chaos can take place in a system that has
only one variable. With continuous time, it can happen in systems with as few as three variables.
6. For given conditions or control parameters, it's entirely self-generated. In other words, changes in other
(i.e. external) variables or parameters aren't necessary.
7. It isn't the result of data inaccuracies, such as sampling error or measurement error. Any particular value
of xt (right or wrong), as long as the control parameter is within an appropriate range, can lead to chaos.
8. In spite of its disjointed appearance, it includes one or more types of order or structure.
9. The ranges of the variables have finite bounds. The bounds restrict the attractor to a certain finite region
in phase space.
10. Details of the chaotic behavior are hypersensitive to changes in initial conditions (minor changes in the
starting values of the variables).
11. Forecasts of long-term behavior are meaningless. The reasons are sensitivity to initial conditions and the
impossibility of measuring a variable to infinite accuracy.
12. Short-term predictions, however, can be relatively accurate.
13. Information about initial conditions is irretrievably lost. In the mathematician's jargon, the equation is
''noninvertible." In other words, we can't determine a chaotic system's prior history.
14. The Fourier spectrum is "broad" (mostly uncorrelated noise) but with some periodicities sticking up here
and there.
15. The phase space trajectory may have fractal properties. (We'll discuss fractals in Ch. 17.)
16. As a control parameter increases systematically, an initially nonchaotic system follows one of a select
few typical scenarios, called routes, to chaos.

7. Attractors – General Statements
An attractor is a dynamical system's set of conditions
In a phase space diagram, an attractor shows a system's long-term behavior. It's a compact, global picture of all of a system's possible steady states.
All attractors are either nonchaotic or chaotic
Nonchaotic attractors generally are points, cycles, or smooth surfaces (tori), and have regular, predictable trajectories…small initial errors or minor perturbations generally don't have significant long-term effects
Chaotic or strange attractors occur only after the onset of chaos. Long term prediction on a chaotic attractor is limited…small initial errors or minor perturbations can have profound long-term effects

8. Strange Attractors
A Strange Attractor is dynamically unstable and non periodic
- A chaotic system is unstable…its behavior changes with time rather than settling to a fixed point
- Chaotic systems are non periodic…trajectories do not settle into repeatable patterns …and never cross
- A chaotic attractor shows extreme sensitivity to initial conditions… trajectories initially close, diverge, and eventually follow very different paths
Summary
An attractor is the set of phase space conditions that represent a system's various possible steady states.
Trajectories generally approach attractors asymptotically and, at least with most mathematical examples, never
get all the way "onto" an attractor. Nonchaotic attractors are of three types:
• the point attractor (a fixed point, therefore occupying zero dimensions in phase space)
• the periodic attractor or "limit cycle" (two or more values that recur in order, occupying two dimensions
in phase space)
• the torus (a combined representation of two or more limit cycles, generally taking the shape of a threedimensional
doughnut in phase space).
A torus can be periodic (meaning that the combined trajectory of the various limit cycles exactly repeats itself)
or quasiperiodic (in which the combined trajectory almost but not exactly repeats itself).

9. Difference between linear and non-linear systems…
• Behavior over time. Linear processes are smooth and regular, whereas nonlinear ones may be regular at first but often change to erratic-looking.
• Response to small changes in the environment or to stimuli. A linear process changes smoothly; the response of a nonlinear system is often much greater than the stimulus.
• Persistence of local pulses. Pulses in linear systems decay over time. In nonlinear systems, pulses can persist for long times, perhaps forever.

10. Non-periodic Dynamical System
A dynamical system that never settles into a steady state attractor
Non periodic systems never settle into a repeatable (predictable) sequence of behavior.
Prediction of a future state of a non periodic system is eventually impossible, due to nonlinear dynamics (feedback)
The atmosphere illustrates non periodic behavior
Broad patterns in the development, evolution, and movement of weather systems may be noticeable, but no patterns ever repeat in an exact and predictable sequence
The atmosphere is:
…damped by friction of moving air and water
…driven by the Sun’s energy
…the ultimate feedback system
Weather patterns never settle into a steady state attractor

11. Sensitivity to Initial Conditions
Small uncertainties (minute errors of measurement which enter into calculations) are amplified
Result: system behavior is predictable in the short term…unpredictable in the long term

12. The Lorenz Discovery
From nearly the same starting point (tiny rounding error), the new forecast diverged from the original forecast…eventually reaching a completely different solution!
Why? …Slight differences in the initial conditions had profound effects on the outcome of the whole system
Lorenz found the mechanism of deterministic chaos: simply- formulated systems with only a few variables can display highly complex and unpredictable behavior
(.506) vs. ( )
Initial condition

13. Chaos and Numerical Weather Prediction (NWP)
If a process is chaotic… knowing when reliable predictability dies out is useful, because predictions for all later times are useless.
Weather forecasts lose skill because of:
Chaos …small errors in the initial state of a forecast grow exponentially
Model uncertainty
Numerical models only approximate the laws of physics (important small scale processes are parameterized)
Very small errors in the initial state of a forecast model grow rapidly at small scales, then spread upscale
Forecast skill varies both spatially and temporally as a result of both initial state and model errors, which change as the atmospheric flow evolves

14. Models must simulate numerous irresolvable processes
A quick overview of the many things a model must forecast that may occur at sub-grid scale resolution. Detailed models can forecast several of these things but it is prohibitive for many models to accomplish this effectively over a large domain.

15. NWP Skill as a Function of Scale and Time
Feature/Variable
Feature/Variable
< Day1
< Day1
Days 1-2
Days 1-2
Days 3-5
Days 3-5
Days 6-7
Days 6-7
Hemispheric flow transitions
Hemispheric flow transitions
Excellent
Excellent
Excellent
Excellent
Very Good
Very Good
Good
Good
Cyclone life cycle
Cyclone life cycle
Excellent
Excellent
Very Good
Very Good
Fair-Good
Fair-Good
Low skill-Fair
Low skill-Fair
Fronts
Fronts
Excellent
Excellent
Good
Good
Fair
Fair
----
----
Mesoscale banded structures
Convective clusters
Mesoscale banded structures
Convective clusters
Good
Good
Fair
Fair
----
----
----
----
Temp / wind
Temp / wind
Excellent
Excellent
Very Good
Very Good
Skill with max/min Temp
Skill with max/min
Precip/ mean clouds
QPF/ mean clouds
Very Good
Very Good
Good
Good
Fair
Some skill in 5-10 day QPF
Predictability falls off as a function of scale
Large scale features (planetary waves) may be predictable up to a week in advance
Small systems (fronts) are well forecasted to day 2.. cyclonic systems to day 4

16. Coping with NWP Predictability
The largest obstacles to realizing the potential predictability of weather and climate are inaccurate models and insufficient observations…an intrinsic limit of predictability will always exist however
In the last 30 years, most improvements in weather forecast skill have come from improvements in models and data assimilation techniques
Even though increased resolution increases error growth rates, increased resolution more accurately simulates physical processes, allowing more accurate scale interactions and forecast evolution
High resolution ensembles represent the best of both worlds:
Added “realism” of high-resolution
An attempt to account for the inherent uncertainties

17. How do Ensembles help us cope with Chaos?

18. Why can’t we count exclusively on single model NWP?
Overlooks forecast uncertainty
Initial condition and model uncertainty
Chaotic flows vs. stable flow regimes
Potentially misleading
Oversells forecast capability

19. GFS 84 hr forecast Valid 00Z 22 Nov
NAM 84 hr forecast Valid 00Z 22Nov
Single Model NWP
Which model do you believe?

20. Ensembles and PDF
High Res Control
Reality
PDF
Time
Single Forecast
Recognizing the eventuality of chaos…weather forecasts can provide more useful information by describing the time evolution of an ensemble probability density function (PDF)
Initial PDF represents initial uncertainty
Single forecast doesn’t account for initial and model error…often fails to predict the real future state past a certain point
Ensemble of perturbed forecasts accounts for initial and model error… PDF of solutions more likely to contain real future state
Ensemble PDF contains additional information, including forecast uncertainties

21. Produce a series of forecasts, each starting from slightly different initial conditions and/or model formulations
Properties of the forecast PDF offer important probabilistic and statistical information…the spread of forecast trajectories quantifies forecast uncertainty
…if the initial perturbations sampled the errors of the day
…and the ensuing ensemble spread captures the forecast errors

22. Ensemble Prediction System (EPS) Goals
Represent initial condition and/or model uncertainty
Determine a range of possible forecast outcomes
Estimate the probability for any individual forecast outcome
General: provide a framework for decision assistance

23. General EPS forecasting tools
Spaghetti Plots (shows all solutions)
Mean/Spread (“middleness” and variability)
Probabilities
Most Likely Event

24. “Spaghetti” Plots

25. Mean and Spread Characteristics of mean Characteristics of spread 4 1
The ensemble mean performs better on average than operational model on which it is based. Why?
Because predictable features remain intact, less predictable features are smoothed out
Characteristics of spread
Allows assessment of uncertainty, since more spread means more uncertainty 4 1 3 2
[start with heading/title only]
[bring in 1st bullet and 1st sub-bullet] In performance scores like the anomaly correlation and the root-mean-square error, the ensemble mean forecast, on average, performs better than operational forecast of even considerably higher resolution. Why is that? Before, we noted that the ensemble mean contour was smoother than the contour for the individual ensemble members. This is [bring in 2nd sub-bullet] because predictable features remain intact in the mean, while less predictable features tend to be smoothed out.
[bring in 2nd bullet and associated sub-bullet]
The spread is useful in that it indicates regions where the atmosphere is less predictable since there is less agreement in the ensemble members.
The graphic shown here [have mean and spread graphic come up] uses the same data as that old unreadable graphic, but rather than showing the contours for all the ensemble members, it shows the ensemble mean, contoured at 60-meter intervals; and spread, in meters, with values indicated by the shading. Note the high values of spread (orange or ‘warmer’ colors) over [bring in circle #1] the west coast of the US, [bring in circle #2] the central plains, [bring in circle #3] the mid-Atlantic coast, and [bring in circle #4] near and east of Labrador.

26. Probability of Exceedance
Helps determine the probability of a specified event.
Gives probability of exceeding meaningful threshold
Calculation represents count of what % of ensemble members exceed the threshold of interest
Example here is for 12-hour precipitation exceeding 0.25 inches.
[start with title only]
[bring in 1st bullet]
Forecasters are paid to predict extreme weather events and other events involving the exceedance of a threshold.
[bring in 2nd bullet]
Ensemble products indicating the probability for exceeding a forecast value help with this task. The probability of exceedance [bring in 1st sub-bullet] is calculated by taking a count of the number of ensemble members that exceed the chosen threshold, and then dividing by the total number of members in the ensemble. [bring in 2nd sub-bullet] Sometimes, an adjustment is made to the probability distribution based on verifications over a period of time before figuring out the probability of exceeding a critical threshold. We’ll talk about that later.
[bring in 3rd bullet]
For example, the graphic below on the right shows the 0000 UTC on 19 November 2001 probability of 24-hour precipitation amounts exceeding 0.5" for the period ending 1200 UTC on 22 November The related spaghetti diagram for 24-hour accumulated precipitation exceeding 0.5" for each ensemble member is shown on the left.

27. Most Likely or Dominant Event Diagram
Used to show what is most often predicted by the ensemble forecast
A common example
Precipitation type (snow, sleet, freezing rain, rain)
[bring in title]
A somewhat related product to the probability of exceedance diagram is the most likely or dominant event diagram. We count items predicting events of concern and then produce a graphic [bring out 1st bullet] showing those most frequently predicted in the ensemble run. The most common example of this is the Dominant Precipitation Type [bring out 2nd bullet and then 1st sub-bullet] graphic, which can be used in combination with probability of exceedance products for precipitation amount for winter weather warnings.
This graphic [bring in graphic] is an example from the NCEP Short-Range Ensemble Prediction system from the 12 UTC 12 November 2002 forecast valid on 00 UTC 15 November Precipitation type is indicated by shading, based on legend in lower left side of graphic. Note that sleet and snow were combined in this graphic into a “snow” precipitation type for purposes of counting.
“Dominant precipitation type” in this case means:
The one that occurs most frequently in all areas where precipitation occurs in at least one ensemble member, as diagnosed by the NCEP precipitation type algorithm.
If there is a tie among two or more ensemble members, a mix of dominant precipitation types is indicated.
In this case, rain is indicated in areas shaded in green [highlight green], snow in areas shaded in blue [highlight blue], freezing rain in areas shown in red [highlight red], purple for snow and freezing rain equally likely [highlight purple well-north of Montana and Washington state], and orange for rain and freezing rain equally likely [highlight orange well-north of Montana].

28. Summary
Chaos and model uncertainties impose a very real physical limit on predictability
Predictability falls off (sometimes rapidly) as a function of scale and time
Forecast accuracy varies both spatially and temporally as a result of initial state and model errors, which change as the atmospheric flow evolves
Ensemble NWP optimizes predictability for all scales, and extends the utility of forecasts…especially at extended ranges (days 4-7)
Allows for quantification of uncertainty, and foundation for decision assistance

29. Statistical Guidance in the Forecast Process

30. WHY STATISTICAL GUIDANCE?
Add value to direct NWP model output
Objectively interpret model
- remove systematic biases
- quantify uncertainty
Predict what the model does not
Produce site-specific forecasts
(i.e. a “downscaling” technique)
Assist forecasters
“First Guess” for expected local conditions
“Built-in” model/climatology

31. MODEL OUTPUT STATISTICS (MOS)
Relates observed weather elements (PREDICTANDS) to appropriate variables (PREDICTORS) via a
statistical approach.
Predictors are obtained from:
Numerical Weather Prediction (NWP) Model
Forecasts
2. Prior Surface Weather Observations
3. Geoclimatic Information
Current Statistical Method:
MULTIPLE LINEAR REGRESSION
(Forward Selection)

32. MODEL OUTPUT STATISTICS (MOS)
Properties
Mathematically simple, yet powerful
Need historical record of observations
at forecast points
(Hopefully a long, stable one!)
Equations are applied to future run of
similar forecast model
Probability forecasts possible from a
single run of NWP model
Other statistical methods can be used
e.g. Polynomial or logistic regression;
Neural networks

33. MODEL OUTPUT STATISTICS (MOS)
ADVANTAGES
- Recognition of model predictability
- Removal of some systematic model bias
- Optimal predictor selection
- Reliable probabilities
- Specific element and site forecasts
DISADVANTAGES
- Short samples
- Changing NWP models
- Availability & quality of observations

39. Now approx sites

41. Gridded MOS “MOS at any point (GMOS)
- Support NWS digital forecast database
2.5 km - 5 km resolution
- Equations valid away from observing sites
- Emphasis on high-density surface networks
- Use high-resolution geophysical data
- Some problems over steep terrain or data-sparse
regions

42. Gridded MOS

43. Use of MOS at a Forecast Office
Can ingest GMOS directly into local digital forecast database
Can apply bias correction (based on performed in past 30 days)
Can ingest point-based MOS and spread it to entire grid
MOS from single models or from ensemble mean/max/min
We verify our forecast against MOS, so we may use as a starting point but we try to improve on it based on local experience or recent trends

44. Questions?