Presentation on theme: "The Seasonal Forecast System"— Presentation transcript:
1 The Seasonal Forecast System at ECMWFTim StockdaleEuropean Centre for Medium-Range Weather Forecasts
2 Contents Seasonal forecasting with coupled GCMs Why use GCMs?How we make a forecastBasic calibration – how, why, and what are the problems?Model errorOperational forecasts: ECMWF System 3System designHow good are the El Nino forecasts?How good are the atmospheric forecasts?Multi-model systems and forecast interpretationEUROSIP multi-model systemMeaning of probabilistic forecasts
3 Sources of seasonal predictability KNOWN TO BE IMPORTANT:El Nino variability - biggest single signalOther tropical ocean SST - important, but multifariousClimate change - especially important in mid-latitudesLocal land surface conditions - e.g. soil moisture in springOTHER FACTORS:Volcanic eruptions - definitely important for large eventsMid-latitude ocean temperatures - still somewhat controversialRemote soil moisture/ snow cover - not well establishedSea ice anomalies - local effects, but remote??Dynamic memory of atmosphere - most likely on 1-2 monthsStratospheric influences - various possibilitiesUnknown or Unexpected - ???
4 Methods of seasonal forecasting Empirical forecastingUse past observational record and statistical methodsWorks with reality instead of error-prone numerical models Limited number of past cases means that it works best when observed variability is dominated by a single source of predictability A non-stationary climate is problematic Two-tier forecast systemsFirst predict SST anomalies (ENSO or global; dynamical or statistical)Use ensemble of atmosphere GCMs to predict global responseMany people still use regression of a predicted El Nino index on a local variable of interestSingle-tier GCM forecastsInclude comprehensive range of sources of predictability Predict joint evolution of SST and atmosphere flow Includes indeterminacy of future SST, important for prob. forecasts Model errors are an issue!
5 Step 1: Build a coupled model IFS (atmosphere)TL159L62 Cy31r1, deg grid for physics (operational in Sep 2006)Full set of singular vectors from EPS system to perturb atmosphere initial conditions (more sophisticated than needed …)Ocean currents coupled to atmosphere boundary layer calculationsHOPE (ocean)Global ocean model, 1x1 mid-latitude resolution, 0.3 near equatorA lot of work in developing the OI ocean analyses, including analysis of salinity, multivariate bias corrections and use of altimetry.CouplingFully coupled, no flux adjustments, except no physical model of sea-ice
6 Step 2: Make some forecasts Initialize coupled system (cf Magdalena’s lecture)Aim is to start system close to reality. Accurate SST is particularly important, plus ocean sub-surface.Don’t worry too much about “imbalances”Run an ensemble forecastExplicitly generate an ensemble on the 1st of each month, with perturbations to represent the uncertainty in the initial conditions; run forecasts for 7 monthsWorry about model error later ….
7 Creating the ensemble Wind perturbations SST perturbations Perfect wind would give a good ocean analysis, but uncertainties are significant. We represent these by adding perturbations to the wind used in the ocean analysis system.BUT only have 5 member ensemble, and no representation of other sources of uncertainty in ocean analysis (obs error, ..)SST perturbationsSST uncertainty is not negligibleSST perturbations added to each ensemble member at start of forecast.BUT perturbations based on analyses that use the same input dataAtmospheric unpredictabilityAtmospheric ‘noise’ soon becomes the dominant source of spread in an ensemble forecast. This sets a fundamental limit to forecast quality.To ensure that noise grows rapidly enough in the first few days, we activate ‘stochastic physics’ and use EPS singular vectors.
8 RMSE and spread in different systems .. but ensemble spread (dashed lines) is still substantially less than actual forecast error.Substantial amounts of forecast error are not from the initial conditions.Rms error of forecasts has been systematically reduced (solid lines) ….
9 Step 3: Remove systematic errors Model drift is typically comparable to signalBoth SST and atmosphere fieldsForecasts are made relative to past model integrationsModel climate estimated from 25 years of forecasts ( ), all of which use a 11 member ensemble. Thus the climate has 275 members.Model climate has both a mean and a distribution, allowing us to estimate eg tercile boundaries.Model climate is a function of start date and forecast lead time.EXCEPTION: Nino SST indices are bias corrected to absolute values, and anomalies are displayed wrt a climate.Implicit assumption of linearityWe implicitly assume that a shift in the model forecast relative to the model climate corresponds to the expected shift in a true forecast relative to the true climate, despite differences between model and true climate.Most of the time, assumption seems to work pretty well. But not always.
10 Plots courtesy Paco Doblas-Reyes Biases (eg JJA 2mT as shown here) are often comparable in magnitude to the anomalies which we seek to predictECMWFMet OfficeMétéo-FranceCFSPlots courtesy Paco Doblas-Reyes
11 SST bias is a function of lead time and season. More recent systems have less bias, but it is still large enough to require correcting for.
12 Despite SST bias and other errors, anomalies in the coupled system can be remarkably similar to those obtained using observed (unbiased) SSTs …..
13 … and can also verify well against observations
14 Model errors are still serious … Models have errors other than mean biasEg weak wind and SST variability in system 2Strong underestimation of MJO activitySuspected too-weak teleconnections to mid-latitudesMean state errors interact with model variabilityNino 4 region is very sensitive (cold tongue/warm pool boundary)Atlantic variability suppressed if mean state is badly wrongOur forecast errors are larger than they should beWith respect to internal variability estimates and (occasionally) other prediction systemsReliability of probabilistic forecasts is not particularly high
15 ENSO forecast quality – Feb. starts only ECMWF S3, 1 Feb Starts, first 3 months of forecastCompletion of TAO arrayDramatic pre/post TAO contrast is specific to these forecasts. Hypothesis is that model error usually dominates forecasts, but at this particular time of year, for this model and this forecast range, model error has negligible impact on forecasts. In this case, the role of initial condition error becomes visible, and the pre/post TAO era is clearly demarcated. At longer lead times (eg May verification), we start to get forecast errors in the post-TAO era, although June and July SSTs verify almost perfectly in the last 10 years ( ).The striking behaviour of the forecasts plotted above was first noted using forecasts to and 2007 (the last two points above) continue the pattern of excellent recent forecasts. The forecast from 2008 looks as if it is also going to be nearly perfect, despite the challenging evolution of SST over the last few months.Note that since the above plot was “selected” from a range of possible plots covering different seasons and lead times, there is likely to be an element of artificial skill. Nonetheless, it is believed that an important part of the apparent error reduction is real.
16 ENSO forecast quality – all start months Model error partially masks the benefit of improved observations
17 System 3 vs Cycle 33r1The most recent ECMWF cycles (such as 33r1, shown here) develop a too-strong cold tongue in the West Pacific. This results in a large drop in skill in SST forecasts in Nino 4. This relationship between mean-state error and forecast performance in the Nino-4 region has been seen time and again in different model versions.The model version used for System 3 has a pretty good balance, but even it is not perfect.
18 Statistical post-processing Red: statistically corrected System 3 forecastImprovement is mainly seen in Nino 4, where there are clear indications of anomaly / mean-state interactions, and where state-dependent errors are visible.(Statistical model is a cross-validated linear regression, based on initial condition SST anomaly and model predicted SST anomaly)Not used operationally
19 Capturing trends is important. Some are handled well ….
20 … other trends are poorly handled Global mean T at 50 hPa. The observed cooling at this altitude is mainly driven by trends in ozone, which our model does not include. The volcanic spikes of El Chichon and Pinatubo are also clear in the observed data.
21 Volcanic influence on Eurasian winters? ECMWF System 3 hindcastAnalysed 2m T anomalyDJF 82/83NH winters with large volcanic aerosol in the NH look like tend to look like 82/83 and 91/92, particularly over Eurasia (based on reconstructions back to 1600). Our coupled forecasts get both these winters badly wrong (although for 91/92, the observed SST does better). Shindell et al 2004 (JGR) have modelling results to suggest volcanic aerosol impact is similar to observed pattern.Even if no volcanoes occur in the future, our estimates of past forecast quality may need to take into account volcanic winters. ECMWF are working on including the relevant aerosol effects (historically, and in the real-time analysis an forecast system).DJF 91/92
22 Operational seasonal forecasts Real time forecasts since 1997“System 1” initially made public as “experimental” in Dec 1997System 2 started running in August 2001, released in early 2002System 3 started running in Sept 2006, operational in March 2007Burst mode ensemble forecastInitial conditions are valid for 0Z on the 1st of a monthForecast is created typically on the 11th/12th (SST data is delayed up to 11 days)Forecast and product release date is 12Z on the 15th.Range of operational productsModerately extensive set of graphical products on webRaw data in MARSFormal dissemination of real time forecast data
23 System 3 configuration Real time forecasts: 41 member ensemble forecast to 7 monthsSST and atmos. perturbations added to each member11 member ensemble forecast to 13 monthsDesigned to give an ‘outlook’ for ENSOOnly once per quarter (Feb, May, Aug and Nov starts)November starts are actually 14 months (to year end)Back integrations from (25 years)11 member ensemble every month5 members to 13 months once per quarter
24 How many back integrations? Back integrations dominate total cost of systemSystem 3: 3300 back integrations (must be in first year)492 real-time integrations (per year)Back integrations define model climateNeed both climate mean and the pdf, latter needs large sampleMay prefer to use a “recent” period (30 years? Or less??)System 2 had a 75 member “climate”, System 3 has 275.Sampling is basically OKBack integrations provide information on skillA forecast cannot be used unless we know (or assume) its level of skillObservations have only 1 member, so large ensembles are much less helpful than large numbers of cases.Care needed eg to estimate skill of 41 member ensemble based on past performance of 11 member ensembleFor regions of high signal/noise, System 3 gives adequate skill estimatesFor regions of low signal/noise (eg <= 0.5), need hundreds of years
25 Example forecast products A few examples only – see web pages for full details and assessment of skillNote: Significance values on plotsA lot of variability in seasonal mean values is due to chaosEnsembles are large enough to test whether any apparent signals are real shifts in the model pdfWe use the w-test, non-parametric, based on the rank distributionNOT related to past levels of skill
40 How good are the forecasts? Deterministic skill: DJF ACCTemperature: actual forecastsTemperature: perfect model
41 How good are the forecasts? Deterministic skill: DJF ACCPrecip: actual forecastsPrecip: perfect model
42 How good are the forecasts? Probabilistic skill: Reliability diagramsTropical precip < lower tercile, JJANH extratrop temp > upper tercile, DJF
43 How good are the forecasts? Probabilistic skill: Reliability diagramsEurope: Temp > upper tercile, DJF
44 EUROSIP multi-model ensemble Three models running at ECMWF:ECMWF – as describedMet Office – HADCM3 model, Met Office ocean analysesMeteo-France – Meteo-France model, Mercator ocean analysesUnified systemReal-time since mid-2005All data in ECMWF operational archiveCommon operational schedule (products released at 12Z on 15th)Common productsCoordinated development strategy (sort of …)See “EUROSIP User Guide” on web for details, and also the recent ECMWF Newsletter article (Issue No. 118, Winter 2008/09)
45 Summer 2009 temperature forecast ECMWFEUROSIP(See Antje’s lecture for more on multi-model ensembles ...)
46 Model error and forecast interpretation Model error is largeIt dominates El Nino forecast errorsMean state and variability errors are very significantErrors cannot be easily fixedProducts typically account for sampling error onlyDon’t take model probabilities as true probabilitiesEstimating forecast skill can be difficultIn many cases, data is insufficient to produce sensible estimatesMakes interpretation of forecasts toughIn the end we need trustworthy models(Multi-model ensembles are small, and only partially span the space of model errors)
47 Calibrated forecasts Frequentist or Bayesian Sampling uncertainties give rise to frequentist probabilitiesModel errors mean our forecast pdfs are wrong. We can try to represent our lack of knowledge as “probabilities” in a Bayesian framework.Calibration of probabilistic forecastsGiven the model forecast pdf, the past history of forecast pdfs, the past history of observed outcomes, what is the best estimate of the true pdf?A Bayesian analysis depends on specification of the prior. Mathematically, no single “right” answer.We hope one day to implement a hierarchical Bayesian analysis, accounting for the major uncertainties and producing some general purpose calibrated probabilistic forecasts.Handling trends and non-stationarities is an additional challenge
48 Some final comments Plenty of scope for improving model forecasts Nino SST forecasts still much worse than predictability limitsModel errors still obvious in many casesInitial conditions probably OK in Pacific for some time, recently improved elsewhere by ARGOModel output -> user forecastCalibration and presentation of forecast informationPotential for multi-model ensemblesTimescale for improvementsOptimist: in 10 years, we’ll have much better models, pretty reliable forecasts, confidence in our ability to handle climate variationsPessimist: in 10 years, modelling will still be a hard problem, and progress will largely be down to improved calibration