Theoretical Review of Seasonal Predictability In-Sik Kang Theoretical Review of Seasonal Predictability In-Sik Kang

Analysis of Variance of JJA Precipitation Anomalies (SNU case) (a) Total variance (b) Forced variance (c) Free variance Free variance Intrinsic transients due to natural variability Forced variance Climate signals caused by external forcing

Forced VarianceInternal Variance Signal-to-noise

Decomposition of climate variables Climate state variable (X) consists of predictable and unpredictable part.  Predictable part = signal (Xs) : forced variability  Unpredictable part = noise (Xn) : internal variability X = Xs + Xn The dynamical forecast (Y) also have its forced and unforced part. forecast signal (Ys) : forced variability of model forecast noise (Yn) : internal variability of model Y = Ys + Yn The internal variability (noise) is stochastic If the forecast model is not perfect, Xs≠Ys. (there is a systematic error)

Prediction skill Noise and Error are not correlated with others Alpha : regression coeff. of signal Cor(x,y)= The correlation coefficient is maximized by removing V(y e ) and V(y n )  The most accurate forecast will be the SIGNAL of perfect model.

When the forecast is perfect signal, the correlation coefficient is Maximum prediction skill : potential predictability Maximum prediction skill (= potential predictability of particular predictand) is a function of Signal to Noise Ratio Cor(x,y)=

Perfect model correlation & Signal to Total variance ratio Z500 winter (C20C, 100 seasons, 4 member) Although the 4 member is not enough to estimate Potential predictability precisely, the patterns of 2 metrics are quite similar

Strategy of prediction 1. Reduction of Noise Averaging large ensemble members (if number of ensemble members is infinte, Noise will be zero in the ensemble mean) 2. Correct signal Improving GCM Statistical post-process (MOS) The strategy of seasonal prediction is to obtain “perfect signal” as close as possible. (i.e. reducing variance of systematic error and variance of noise)

Ensemble averaging : reduction of noise Necessary number of ensemble is dependent on the signal to noise ratio Extratropical forecast needs larger ensemble members than tropics. Correlation with perfect forecast (perfect signal) N : number of ensemble member

Multi Model Ensemble The averaging of large ensembles reduces noise in the forecast. The multi model ensemble combines ensembles with different forecast signal, the cancellation of systematic bias is possible : a sort of post processing. Thus the multi model ensemble (MME) can be more efficient ensemble technique to get “perfect signal” : it permits both of noise reduction and signal correction. And statistically optimized MME technique (eg. superensemble) can be more beneficial in the correction of systematic error like as usual statistical post-processes. However, there are still debates on benefit of multi model ensemble in seasonal prediction.

□ Pattern Correlation (0-360E, 40S-60N) MME1 MME2 MME3 Single model (a) 850 hPa Temperature (b) Precipitation 0.43, 0.44, , 0.47, 0.58 MMES (based on point-wise correction, CPPM)

Issues on Multi Model Ensemble prediction Debates on MMEP of Seasonal forecast Is a multi model system better than a single good model? (Graham et al. 2000; Peng et al. 2002; Doblas-Reyes et al. 2000) Is a sophisticated technique better than a simple composite? (Krishnamurti et al. 2000; Kharin and Zwiers 2002; Pavan and Doblas-Reyes 2000 ) Strong limitation of seasonal predictability study : small samples MME prediction experiment in a simple climate system (Krishnamurti et al. 2000; Palmer 1993, 1999; Qin and Robinson 1995)

Forced variability Internal variability  Design of the simple climate model and predictability experiment Simple Chaotic Model Low frequency forcing Atmospheric process Simple Chaotic Model 1 Simple Chaotic Model 2 Simple Chaotic Model 9 : Different parameters  Simple chaotic model : 2 layer Nonlinear QG spectral model, Reinhold & Pierrehumbert (1982) Time varying forcing (Interannual time scale) Multimodel ensemble forecast in simple model Equivalent to the seasonal forecast using multi-AGCMs with prescribed SST

Model parameter M1M2M3M4M5M6M7M8Obs k’k’ h ’’ Multimodel ensemble forecast in simple model Physical parameters of each models Observation Case 1 Case 2 Prediction (20 ensembles) Ens. mean Prediction experiments : 120cases, 20 ensembles

Interannual variability : a particular wave component Obs model Fcst (ensemble member)

Multimodel ensemble prediction schemes MME1 : simple composite of individual forecast with equal weighting. (special case of MME2) MME2 (Superensemble) : Optimally weighted composite of individual forecasts. The weighting coefficient is defined by regression of forecasts and observation during training period. MME3 : simple composite of individual forecasts, which was corrected by statistical post process Observation Hindcast ■ Statistical correction (Kang et al. 2004) SVD Coupled Pattern New Forecast Corrected forecast Projection coeff.

Prediction skills of single and multi model ensemble Pattern correlation of indiv. Model and MME (60case avg.) corrected MME1 MME2 MME3 MME is not better than a single good model !! It is due to the inclusion of bad model (M2, M6) What will happen if bad model excluded in MME?

Which models? : combination of models skillModels , 2, 3, 4, 5, 6, 7, , 2, 3, 4, 5, 6, 7, , 2, 3, 4, 5, 6, 7, 8 skillModels , 2, 3, 4, 5, 6, 7, , 2, 3, 4, 5, 6, 7, , 2, 3, 4, 5, 6, 7, 8 Good combinations (MME1)Bad combinations (MME1) Combining all available models does not guarantee best skill “Systematically” bad model needs to be excluded. skillModels , 2, 3, 4, 5, 6, 7, , 2, 3, 4, 5, 6, 7, , 2, 3, 4, 5, 6, 7, 8 skillModels , 2, 3, 4, 5, 6, 7, , 2, 3, 4, 5, 6, 7, , 2, 3, 4, 5, 6, 7, 8 Good combinations (MME2)Bad combinations (MME2) “Systematically” bad model can be useful : difficult to find good combination Comparing 219 combinations (3 to 8 models)

Number of models MME1 MME2 MME3 NT=30 NT=60 NT=90  Composite forecast : more models, better forecasts  Due to overfitting, regression based forecast getting worse with increasing number of models.  When the signal to noise ratio is large, superensemble is more skillful than simple composite. 30 case mean skill averaged over each number of models

Debates on MMEP of Seasonal forecast Is a multi model system better than a single good model? It depends on the combination of models, if there are sysetmatically bad model, MME is not better than a single good model. Is a sophisticated technique better than a simple composite? When the signal to noise ratio is small, superensemble tends to be unstable due to overfitting. On the other hand, superensemble can be better than a simple composite where the signal to noise ratio ( potential predictability is high) and number of model is not large. Results of simple model experiments

Summary Due to the unpredictable noise, the most accurate deterministic forecast is a perfect signal. (ensemble mean of perfect model) To obtain perfect signal, we have to - reduce noise in the forecast - correct forecast signal (systematic error correction) MME is a reasonable approach to the perfect signal Composite based MME has some dependency on the combination of models : need to exclude bad models. Complex MME (Superensemble) have a problem of overfitting in the case of low signal to noise ratio and short historical record. Generally, Composite based MME is more feasible and skillful. The composite of calibrated forecast is more beneficial.

Probabilistic Seasonal Forecasts and the Economic value

Value of the forecast : accuracy & utility Forecast is valuable when it is used by decision maker and has some benefits. Forecast Information $, ¥, ₩,  Decision making Resolution → Utility Accuracy Forecast value Resolution → Forecast value = Accuracy x Utility

Choice of forecast system Decision maker Forecast system Choice : the most Valuable forecast system The form and properties should be matched by a particular situation of user  Climatological probability of event : P c  Cost-Loss ratio of user : C/L  Flexibility of interpretation by user : P t

OBS Single model Multi model Probability distribution of summer mean precipitation □ Distribution of total ensemble members

Probability formulation Climatological PDF Ensemble PDF of particular year 0 Xc -Xc A B C A B C Probability of ABOVE normal Probability of NORMAL Probability of BELOW normal

Multi model ensemble probability Model 1Model 2 Collecting all normalized ensemble members (5 model, 45 samples) MME1 Probability MME3 Probability Change the ensemble mean with MME3 in deterministic forecast μ μ μ*μ* μ : ensemble mean μ *: corrected ensemble mean Model 3 MME1 Value Number of occurrence

Reliability Diagram (Above normal) (a) Monsoon(40E-160E,20S~60N)(b) ENSO (160E-280E,20S~20N) Single model MME1 MME3 Fcst probability Obs. probability

Reliability Diagram (Below normal) (a) Monsoon(40E-160E,20S~60N)(b) ENSO (160E-280E,20S~20N) Fcst probability Obs. probability Single model MME1 MME3

Economic Value of Prediction System CONTINGENCY TABLE OF C/L Hit (h) Mitigated loss (C+Lu) Miss (m) Loss (L=Lp+Lu) False Alarm (f) Cost (C) Correct rejection (c) No cost (N) Forecast/action Observation YesNo Yes No C: Total cost r=C/Lp Lu: Unprotectable loss o: the climatetological Lp: Protected against frequency of the event ECONOMIC VAULE : V Global East Asia West US Australia Economic Value Economic Value as a Function of Cost/Loss Ratio (GCPS)

Single model MME1 MME2 MME3 Economic value of deterministic forecast (Above normal) (a) Monsoon(40E-160E,20S~60N)(b) ENSO (160E-280E,20S~20N) C/L

Single model MME1 MME3 (a) Monsoon(40E-160E,20S~60N)(b) ENSO (160E-280E,20S~20N) Economic value of Probabilistic forecast (Above normal) Black line: MME3 in deterministic forecast

Economic value of forecast (above normal, C/L=0.1) DeterministicProbabilistic

Generalized application of forecast  Application to the simple decision system : Y or N – 1bit problem. Observation (real event) YesNo Forecast (action) YesCost NoLoss0 Probabilistic forecast is converted deterministic forecast using p t Benefit of probabilistic forecast cannot be maximized.  Application to the generalized decision system : Action function & Contingency map Temperature (T) Action function : F(T) Probability forecast p(T) Deterministic forecast (T 0 ) Action using Det. Forecast : F(T 0 ) Action using Prob. Forecast : ∫p(T)F(T)dT Forecast utilization covering wide range of problem Forecast Observation Net benefit  Contingency map  Action function