Steve Keighton National Weather Service Blacksburg, VA Predictability, Ensemble Forecasts, and the use of Statistical Guidance in the Forecast Process Steve Keighton National Weather Service Blacksburg, VA

Outline Acknowledgments Chaos theory and predictability in the atmosphere Numerical Weather Prediction (NWP) and use of “ensemble” forecast methods Use of statistical guidance in the forecast process (Model Output Statistics) – if time Acknowledgments Josh Korotky – NWS Pittsburgh Mark Antolik – NWS Meteorological Development Lab

“Prediction is very difficult, especially about the future” “Prediction is very difficult, especially about the future” - Niels Bohr

Prelude – What is Chaos and why is it important? Chaos leads us from the laws of nature to their consequences …shows us that simple systems can exhibit complex behavior…and vice versa …demonstrates that unpredictable behavior can develop in a system governed by deterministic laws As forecasters…chaos shows us the limits of predictability …highlights the importance of probabilistic thinking …shows us the value of expressing uncertainty in forecasts …helps us understand why the future of forecasting will lean heavily on ensemble rather than deterministic approaches

Edward Lorenz (1917 – 2008) Small errors in the initial state estimate of a nonlinear system can limit the prediction of later states of the system Chaos occurs when error propagation, seen as a signal in time, grows to the same size or scale as the original signal... “… one flap of a sea-gull’s wing may forever change the future course of the weather” (Lorenz, 1963)

Elements of Chaos Dynamical system – future states caused by past states (determinism) Nonlinearity – system output (response) isn't proportional to input (forcing)… a small forcing can lead to a disproportionately large response and vise versa a system's values at one time aren‘t proportional to the values at an earlier time Non-periodic behavior – future states never repeat past states Extreme sensitivity to initial conditions – small initial state uncertainties amplify… a "prediction horizon” is inevitable Even though the governing laws of a system are known, long-term predictions can be meaningless Chaos occurs only in deterministic, nonlinear, dynamical systems Chaos results from a deterministic process. 2. It happens only in nonlinear systems. 3. The motion or pattern for the most part looks disorganized and erratic, although sustained. In fact, it can usually pass all statistical tests for randomness. 4. It happens in feedback systems—systems in which past events affect today's events, and today's events affect the future. 5. It can result from relatively simple systems. With discrete time, chaos can take place in a system that has only one variable. With continuous time, it can happen in systems with as few as three variables. 6. For given conditions or control parameters, it's entirely self-generated. In other words, changes in other (i.e. external) variables or parameters aren't necessary. 7. It isn't the result of data inaccuracies, such as sampling error or measurement error. Any particular value of xt (right or wrong), as long as the control parameter is within an appropriate range, can lead to chaos. 8. In spite of its disjointed appearance, it includes one or more types of order or structure. 9. The ranges of the variables have finite bounds. The bounds restrict the attractor to a certain finite region in phase space. 10. Details of the chaotic behavior are hypersensitive to changes in initial conditions (minor changes in the starting values of the variables). 11. Forecasts of long-term behavior are meaningless. The reasons are sensitivity to initial conditions and the impossibility of measuring a variable to infinite accuracy. 12. Short-term predictions, however, can be relatively accurate. 13. Information about initial conditions is irretrievably lost. In the mathematician's jargon, the equation is ''noninvertible." In other words, we can't determine a chaotic system's prior history. 14. The Fourier spectrum is "broad" (mostly uncorrelated noise) but with some periodicities sticking up here and there. 15. The phase space trajectory may have fractal properties. (We'll discuss fractals in Ch. 17.) 16. As a control parameter increases systematically, an initially nonchaotic system follows one of a select few typical scenarios, called routes, to chaos.

Attractors – General Statements An attractor is a dynamical system's set of conditions In a phase space diagram, an attractor shows a system's long-term behavior. It's a compact, global picture of all of a system's possible steady states. All attractors are either nonchaotic or chaotic Nonchaotic attractors generally are points, cycles, or smooth surfaces (tori), and have regular, predictable trajectories…small initial errors or minor perturbations generally don't have significant long-term effects Chaotic or strange attractors occur only after the onset of chaos. Long term prediction on a chaotic attractor is limited…small initial errors or minor perturbations can have profound long-term effects

Strange Attractors A Strange Attractor is dynamically unstable and non periodic - A chaotic system is unstable…its behavior changes with time rather than settling to a fixed point - Chaotic systems are non periodic…trajectories do not settle into repeatable patterns …and never cross - A chaotic attractor shows extreme sensitivity to initial conditions… trajectories initially close, diverge, and eventually follow very different paths Summary An attractor is the set of phase space conditions that represent a system's various possible steady states. Trajectories generally approach attractors asymptotically and, at least with most mathematical examples, never get all the way "onto" an attractor. Nonchaotic attractors are of three types: • the point attractor (a fixed point, therefore occupying zero dimensions in phase space) • the periodic attractor or "limit cycle" (two or more values that recur in order, occupying two dimensions in phase space) • the torus (a combined representation of two or more limit cycles, generally taking the shape of a threedimensional doughnut in phase space). A torus can be periodic (meaning that the combined trajectory of the various limit cycles exactly repeats itself) or quasiperiodic (in which the combined trajectory almost but not exactly repeats itself).

Difference between linear and non-linear systems… • Behavior over time. Linear processes are smooth and regular, whereas nonlinear ones may be regular at first but often change to erratic-looking. • Response to small changes in the environment or to stimuli. A linear process changes smoothly; the response of a nonlinear system is often much greater than the stimulus. • Persistence of local pulses. Pulses in linear systems decay over time. In nonlinear systems, pulses can persist for long times, perhaps forever.

Non-periodic Dynamical System A dynamical system that never settles into a steady state attractor Non periodic systems never settle into a repeatable (predictable) sequence of behavior. Prediction of a future state of a non periodic system is eventually impossible, due to nonlinear dynamics (feedback) The atmosphere illustrates non periodic behavior Broad patterns in the development, evolution, and movement of weather systems may be noticeable, but no patterns ever repeat in an exact and predictable sequence The atmosphere is: …damped by friction of moving air and water …driven by the Sun’s energy …the ultimate feedback system Weather patterns never settle into a steady state attractor

Sensitivity to Initial Conditions Small uncertainties (minute errors of measurement which enter into calculations) are amplified Result: system behavior is predictable in the short term…unpredictable in the long term

The Lorenz Discovery From nearly the same starting point (tiny rounding error), the new forecast diverged from the original forecast…eventually reaching a completely different solution! Why? …Slight differences in the initial conditions had profound effects on the outcome of the whole system Lorenz found the mechanism of deterministic chaos: simply- formulated systems with only a few variables can display highly complex and unpredictable behavior (.506) vs. (.506127) Initial condition

Chaos and Numerical Weather Prediction (NWP) If a process is chaotic… knowing when reliable predictability dies out is useful, because predictions for all later times are useless. Weather forecasts lose skill because of: Chaos …small errors in the initial state of a forecast grow exponentially Model uncertainty Numerical models only approximate the laws of physics (important small scale processes are parameterized) Very small errors in the initial state of a forecast model grow rapidly at small scales, then spread upscale Forecast skill varies both spatially and temporally as a result of both initial state and model errors, which change as the atmospheric flow evolves

Models must simulate numerous irresolvable processes A quick overview of the many things a model must forecast that may occur at sub-grid scale resolution. Detailed models can forecast several of these things but it is prohibitive for many models to accomplish this effectively over a large domain.

NWP Skill as a Function of Scale and Time Feature/Variable Feature/Variable < Day1 < Day1 Days 1-2 Days 1-2 Days 3-5 Days 3-5 Days 6-7 Days 6-7 Hemispheric flow transitions Hemispheric flow transitions Excellent Excellent Excellent Excellent Very Good Very Good Good Good Cyclone life cycle Cyclone life cycle Excellent Excellent Very Good Very Good Fair-Good Fair-Good Low skill-Fair Low skill-Fair Fronts Fronts Excellent Excellent Good Good Fair Fair ---- ---- Mesoscale banded structures Convective clusters Mesoscale banded structures Convective clusters Good Good Fair Fair ---- ---- ---- ---- Temp / wind Temp / wind Excellent Excellent Very Good Very Good Skill with max/min Temp Skill with max/min Precip/ mean clouds QPF/ mean clouds Very Good Very Good Good Good Fair Some skill in 5-10 day QPF Predictability falls off as a function of scale Large scale features (planetary waves) may be predictable up to a week in advance Small systems (fronts) are well forecasted to day 2.. cyclonic systems to day 4

Coping with NWP Predictability The largest obstacles to realizing the potential predictability of weather and climate are inaccurate models and insufficient observations…an intrinsic limit of predictability will always exist however In the last 30 years, most improvements in weather forecast skill have come from improvements in models and data assimilation techniques Even though increased resolution increases error growth rates, increased resolution more accurately simulates physical processes, allowing more accurate scale interactions and forecast evolution High resolution ensembles represent the best of both worlds: Added “realism” of high-resolution An attempt to account for the inherent uncertainties

How do Ensembles help us cope with Chaos?

Why can’t we count exclusively on single model NWP? Overlooks forecast uncertainty Initial condition and model uncertainty Chaotic flows vs. stable flow regimes Potentially misleading Oversells forecast capability

GFS 84 hr forecast Valid 00Z 22 Nov NAM 84 hr forecast Valid 00Z 22Nov Single Model NWP Which model do you believe?

Ensembles and PDF High Res Control Reality PDF Time Single Forecast Recognizing the eventuality of chaos…weather forecasts can provide more useful information by describing the time evolution of an ensemble probability density function (PDF) Initial PDF represents initial uncertainty Single forecast doesn’t account for initial and model error…often fails to predict the real future state past a certain point Ensemble of perturbed forecasts accounts for initial and model error… PDF of solutions more likely to contain real future state Ensemble PDF contains additional information, including forecast uncertainties

Produce a series of forecasts, each starting from slightly different initial conditions and/or model formulations Properties of the forecast PDF offer important probabilistic and statistical information…the spread of forecast trajectories quantifies forecast uncertainty …if the initial perturbations sampled the errors of the day …and the ensuing ensemble spread captures the forecast errors

Ensemble Prediction System (EPS) Goals Represent initial condition and/or model uncertainty Determine a range of possible forecast outcomes Estimate the probability for any individual forecast outcome General: provide a framework for decision assistance

General EPS forecasting tools Spaghetti Plots (shows all solutions) Mean/Spread (“middleness” and variability) Probabilities Most Likely Event

“Spaghetti” Plots

Mean and Spread Characteristics of mean Characteristics of spread 4 1 The ensemble mean performs better on average than operational model on which it is based. Why? Because predictable features remain intact, less predictable features are smoothed out Characteristics of spread Allows assessment of uncertainty, since more spread means more uncertainty 4 1 3 2 [start with heading/title only] [bring in 1st bullet and 1st sub-bullet] In performance scores like the anomaly correlation and the root-mean-square error, the ensemble mean forecast, on average, performs better than operational forecast of even considerably higher resolution. Why is that? Before, we noted that the ensemble mean contour was smoother than the contour for the individual ensemble members. This is [bring in 2nd sub-bullet] because predictable features remain intact in the mean, while less predictable features tend to be smoothed out. [bring in 2nd bullet and associated sub-bullet] The spread is useful in that it indicates regions where the atmosphere is less predictable since there is less agreement in the ensemble members. The graphic shown here [have mean and spread graphic come up] uses the same data as that old unreadable graphic, but rather than showing the contours for all the ensemble members, it shows the ensemble mean, contoured at 60-meter intervals; and spread, in meters, with values indicated by the shading. Note the high values of spread (orange or ‘warmer’ colors) over [bring in circle #1] the west coast of the US, [bring in circle #2] the central plains, [bring in circle #3] the mid-Atlantic coast, and [bring in circle #4] near and east of Labrador.

Probability of Exceedance Helps determine the probability of a specified event. Gives probability of exceeding meaningful threshold Calculation represents count of what % of ensemble members exceed the threshold of interest Example here is for 12-hour precipitation exceeding 0.25 inches. [start with title only] [bring in 1st bullet] Forecasters are paid to predict extreme weather events and other events involving the exceedance of a threshold. [bring in 2nd bullet] Ensemble products indicating the probability for exceeding a forecast value help with this task. The probability of exceedance [bring in 1st sub-bullet] is calculated by taking a count of the number of ensemble members that exceed the chosen threshold, and then dividing by the total number of members in the ensemble. [bring in 2nd sub-bullet] Sometimes, an adjustment is made to the probability distribution based on verifications over a period of time before figuring out the probability of exceeding a critical threshold. We’ll talk about that later. [bring in 3rd bullet] For example, the graphic below on the right shows the 0000 UTC on 19 November 2001 probability of 24-hour precipitation amounts exceeding 0.5" for the period ending 1200 UTC on 22 November 2001. The related spaghetti diagram for 24-hour accumulated precipitation exceeding 0.5" for each ensemble member is shown on the left.

Most Likely or Dominant Event Diagram Used to show what is most often predicted by the ensemble forecast A common example Precipitation type (snow, sleet, freezing rain, rain) [bring in title] A somewhat related product to the probability of exceedance diagram is the most likely or dominant event diagram. We count items predicting events of concern and then produce a graphic [bring out 1st bullet] showing those most frequently predicted in the ensemble run. The most common example of this is the Dominant Precipitation Type [bring out 2nd bullet and then 1st sub-bullet] graphic, which can be used in combination with probability of exceedance products for precipitation amount for winter weather warnings. This graphic [bring in graphic] is an example from the NCEP Short-Range Ensemble Prediction system from the 12 UTC 12 November 2002 forecast valid on 00 UTC 15 November 2002. Precipitation type is indicated by shading, based on legend in lower left side of graphic. Note that sleet and snow were combined in this graphic into a “snow” precipitation type for purposes of counting. “Dominant precipitation type” in this case means: The one that occurs most frequently in all areas where precipitation occurs in at least one ensemble member, as diagnosed by the NCEP precipitation type algorithm. If there is a tie among two or more ensemble members, a mix of dominant precipitation types is indicated. In this case, rain is indicated in areas shaded in green [highlight green], snow in areas shaded in blue [highlight blue], freezing rain in areas shown in red [highlight red], purple for snow and freezing rain equally likely [highlight purple well-north of Montana and Washington state], and orange for rain and freezing rain equally likely [highlight orange well-north of Montana].

Summary Chaos and model uncertainties impose a very real physical limit on predictability Predictability falls off (sometimes rapidly) as a function of scale and time Forecast accuracy varies both spatially and temporally as a result of initial state and model errors, which change as the atmospheric flow evolves Ensemble NWP optimizes predictability for all scales, and extends the utility of forecasts…especially at extended ranges (days 4-7) Allows for quantification of uncertainty, and foundation for decision assistance

Statistical Guidance in the Forecast Process

WHY STATISTICAL GUIDANCE? Add value to direct NWP model output Objectively interpret model - remove systematic biases - quantify uncertainty Predict what the model does not Produce site-specific forecasts (i.e. a “downscaling” technique) Assist forecasters “First Guess” for expected local conditions “Built-in” model/climatology

MODEL OUTPUT STATISTICS (MOS) Relates observed weather elements (PREDICTANDS) to appropriate variables (PREDICTORS) via a statistical approach. Predictors are obtained from: Numerical Weather Prediction (NWP) Model Forecasts 2. Prior Surface Weather Observations 3. Geoclimatic Information Current Statistical Method: MULTIPLE LINEAR REGRESSION (Forward Selection)

MODEL OUTPUT STATISTICS (MOS) Properties Mathematically simple, yet powerful Need historical record of observations at forecast points (Hopefully a long, stable one!) Equations are applied to future run of similar forecast model Probability forecasts possible from a single run of NWP model Other statistical methods can be used e.g. Polynomial or logistic regression; Neural networks

MODEL OUTPUT STATISTICS (MOS) ADVANTAGES - Recognition of model predictability - Removal of some systematic model bias - Optimal predictor selection - Reliable probabilities - Specific element and site forecasts DISADVANTAGES - Short samples - Changing NWP models - Availability & quality of observations

Now approx. 1820 sites

Gridded MOS “MOS at any point (GMOS) - Support NWS digital forecast database 2.5 km - 5 km resolution - Equations valid away from observing sites - Emphasis on high-density surface networks - Use high-resolution geophysical data - Some problems over steep terrain or data-sparse regions

Gridded MOS

Use of MOS at a Forecast Office Can ingest GMOS directly into local digital forecast database Can apply bias correction (based on performed in past 30 days) Can ingest point-based MOS and spread it to entire grid MOS from single models or from ensemble mean/max/min We verify our forecast against MOS, so we may use as a starting point but we try to improve on it based on local experience or recent trends

Questions?