CTB Science Plan For Multi Model Ensembles (MME) Suru Saha Environmental Modeling Centre NCEP/NWS/NOAA

Vision A multi model ensemble forecast system that leverages the best national and international models for improved predictions on intraseasonal, interannual and decadal time scales

Background Studies have shown that the forecast skill of a multi- model system is higher than that of most of the individual models To maximize the amount of model information for NOAA’s seasonal forecasts, a multi-model ensemble strategy will be pursued by the CTB

Current Status of IMME There has been progress on a possible International MME (IMME) with the CFS and the EUROSIP models (ECMWF, UKMO and MeteoFrance) Preliminary discussions have taken place between NCEP and ECMWF on protocols for the transfer of the hindcast data, as well as real time forecasts. Each center will make available new hindcasts, if and when, they upgrade their seasonal forecast system.

Current Status of NMME There has been little progress on a possible National MME (NMME) with the CFS and the NCAR, NASA and GFDL models Results from a previous version of the GFDL model and data assimilation system did not show significant additional skill to the CFS, for the basic metrics of US surface temperature and precipitation.

Current Status of NMME (contd) GFDL has made some new hindcasts. Unfortunately, the CTB has no resources to evaluate these hindcasts. Regarding the NCAR model, Ben Kirtman has been funded to make the hindcasts using the same strategy as the CFSRR Regarding the NASA model, there has been no update on the completion of their hindcasts.

Gaps There are three major gaps in executing a NMME strategy Firstly, skill levels for all models are barely above climatological probabilities for US surface temps and precipitation, at monthly and seasonal time scales. Unless there is a significant improvement in these metrics, the value of a NMME may not be sufficient to justify the major cost of making hindcasts to support an operational NMME.

Gaps (contd) Secondly, the computational and human costs of preparing hindcasts for evaluation of skill and calibration is significant, and not presently funded by NOAA for models other than the CFS. Thirdly, the cost and logistics of running a NMME for all models, in operations, are cosiderable and has not yet been supported by NOAA.

Future Strategy of the NMME (Option 1) With less than a 100% investment, including modest computer resources and a staff of 1 full time and 2-3 part time personnel, the CTB could support the current strategy of: All participating centers make hindcasts of 2 initial months (for the mid-May and mid-November release of CPC’s official monthly and seasonal forecast) over the period 1979-present (or more years, if possible).

Future Strategy of the NMME (Option 1) A dataset of prescribed output variables from these hindcasts (full fields) will be made available to the CTB consolidation team. They will make an objective evaluation of these monthly and seasonal hindcasts, to determine if skill is added to the CFS for the basic metrics of US T & P. Systematic error correction will be done for each tool separately, and the regression coefficients will be determined under a three- years-out Cross-Validation approach. If the weight assigned to a model is near zero, then the consolidation will be redone without that model and the originators of the model will be notified.

Future Strategy of the NMME (Option 1) contd If there is additional skill to the CFS, the participating center will work with EMC in porting the frozen model and data assimilation systems to NCEP. Random hindcasts will be made using this system to ensure that there is reproducibility of the results on the NCEP operational/development computers. This is important for the application of calibration and weights for consolidation (which are derived from the retrospective forecasts) to real time forecasts which will be made on the NCEP computer. This process will involve computer and human resources, which will need to be funded.

Future Strategy of the NMME (Option 1) contd If the results are reproducible, then the participating center will complete the rest of the 10 calendar months with the exact same frozen system. They will make the same datasets available to the CTB consolidation team, for further evaluation to determine if skill is being added to the CFS for these months. If there is additional skill to the CFS, then EMC will work with NCO (computer operations) to implement the NMME into operations at NCEP.

Future Strategy of the NMME (Option 2) With 100% investment, including a dedicated NOAA mainframe computer, with a huge number of nodes, disk space, tapes, and a support staff on the order of 20 to 25 full time personnel, CTB could support porting all participating systems (including models and assimilation systems) to the CTB computer. CTB would then proceed with making the hindcasts as described above. This could be done with individual models or with combinations of model physics and numerics within the Earth System Modeling Framework (ESMF) to form a true NMME.

Future Strategy of the NMME (Option 2) contd The seven year cycle Start CFS Reanalysis at same time as porting all the other models (GFDL, NCAR, NASA) to the NOAA NMME computer. Make sure Reanalysis (CFS) initial conditions can be used in all these models (GFDL, NCAR, NASA), or make Reanalysis for each model system. This implies that improvements that have been made to these models and data assimilation systems every 7 years will be utilized in the NMME, in a continuous fashion, hopefully leading to an increase in skill of the NMME every time.

Future Strategy of the NMME (Option 2) contd The seven year cycle Make hindcasts (2 months, mid-May and mid-November release) for each model system separately or in an optimal combination of different physics parameterizations and/or different dynamic cores Evaluate to see if NMME has additional skill to the CFS (CTB Consolidation team) If additonal skill exists, make hindcasts for remaining 10 calendar months Redo evaluation for additional skill to the CFS for these months If additional skill exists, then transition to NCEP operations.

Why the Need for Long Hindcast Data for S/I prediction ? Huug van den Dool (CPC) and Suranjana Saha (EMC) National Centers for Environmental Prediction, NWS/NOAA/DOC

“Statistical correction of today’s numerical forecasts using a long set of reforecasts (hindcasts) dramatically improves its forecast skill” (Hamill, Whitaker and Mullen 2006, BAMS) (Single model, multi-membered ensemble, NWP)

Data Used for this study DEMETER : 7 European Coupled GCMs CFS (21 years of hindcasts) 4 initial months Monthly Data Focus: Skill in Tmp2m over US lead 3 February forecasts from November initial conditions

Explained Variance (%) for US monthly T2m Feb ; lead 3 (Nov starts) Verification : Climate Division data) SE corr CFSECMPLAMET FRA UKMINGVLODCER 0 years years years 11.2 (0.33 corr) The 3 best models are operational models Skill Estimates for 8-years are not stable Best skill obtained for all 21 years

SE corr CFSECMPLAMET FRA UKM O INGVLOD YC CER F MME (EW) 0 years years years (all) Explained Variance (%) for US monthly T2m Feb ; lead 3 (Nov starts) Verification : Climate Division data) Skill of MME with equal weights for ALL models is far less than the skill of the 3 best models

Explained Variance (%) for US monthly T2m Feb ; lead 3 (Nov starts) Verification : Climate Division data) SE corr CFSECMPLAMET FRA UKMINGLODCERMME (EW) MME3 0 years 2.1 % years years (corr= 51%) Skill of MME with equal weights for 3 best models improves with 21 years

Need more years to determine the SE where/when interannual stand dev is large 8-year 21-year 8yr minus 21yr CFS US TMP2M LEAD 3 FORECAST VERIFYING IN FEB

8-year 21-year MME, ALL MODELS, EQUAL WEIGHT US TMP2M LEAD 3 FORECAST VERIFYING IN FEB 8yr minus 21yr Skill of MME is very much less than CFS when all models are included

MME, 3 BEST MODELS, EQUAL WEIGHT US TMP2M LEAD 3 FORECAST VERIFYING IN FEB 8-year21-year 8yr minus 21yr 8yr minus 21-year Skill of MME for 3 best models is the most for 21 years

Explained Variance (%) for US monthly Precip Feb ; lead 3 (Nov starts) Verification : Climate Division data) SE corr CFSECMPLAMET FRA UKMINGVLODCER 0 years NA years NA years NA Not much Skill in any of the models for US precipitation

15-member CFS reforecasts “ONLY 1 BIG ENSO EVENT” ALL ENSO EVENTS

CONCLUSIONS Without SEC (systematic error correction) there is no skill by any method (for presumably the best month: Feb) With SEC (1 st moment only), there is skill by only a few models (5 out of 8 are still useless) MME not good when quality of models varies too much MME3 works well, when using just three good models

CONCLUSIONS (contd) CFS improves the most from extensive hindcasts (21 years noticeably better than 8) and has the most skill. Other models have far less skill with all years included. Cross validation (CV) is problematic (leave 3 years out when doing 8 year based SEC?) Need more years to determine the SEC where/when the inter annual standard deviation is large