1. Bob Livezey and Marina Timofeyeva
Skill Variability Diagnosis for Discriminating Use of CPC Long-Lead Seasonal Forecasts
Bob Livezey and Marina Timofeyeva
NOAA/NWS/OCWWS/Climate Services Division
Climate Prediction Applications Science Workshop
March 9-11, 2004
Tallahassee, FL

2. Outline Introduction CPC Skill Graphs and Some Other Stratifications
Results
Conclusions and Lessons

3. Seasonal Temperature Forecast Skill 1960s to 80s
Introduction
Seasonal Temperature Forecast Skill 1960s to 80s
All Seasons 8.3
Winter
Spring
Summer
Fall
Users should only care about the performance of forecasts that can potentially benefit their decision process
Livezey (1990): There are non-random subsets of seasonal forecasts that were skillful enough to be useful
After a common presumption was that forecasts are generally skillful enough to be useful

4. Introduction (Cont.) This talk will make the point :
That was made by Livezey (1990)
That there are many non-random subsets of forecasts that do not have useful skill
That it is critical for this information to be shared with potential users
That skill analyses with different stratifications are highly informative while CPC’s web displays are not

5. Displays and Stratifications
CPC Seasonal Forecasts
For 3-equally probable temperature and precipitation classes at 102 Climate Divisions
Made every month from 1995 to present for 0.5-, 1.5-, …, 12.5 month leads
Skill Measure: Modified Heidke Skill Score of Categorized Forecasts
Displays and Stratifications
CPC: Summed over all forecasts for each lead and displayed with times series for this lead
Here:
Summed over all forecasts for each lead and all leads displayed together
Stratified further by cold seasons (DJF to FMA) and warm seasons (MAM to NDJ)
Stratified further by strong ENSO years vs. other years
Stratified by region

6. Displays and Stratifications
Stratification by Lead and Seasons: Temperature
CPC Seasonal Forecasts
For 3-equally probable temperature and precipitation classes at 102 Climate Divisions
Made every month from 1995 to present for 0.5-, 1.5-, …, 12.5 month leads
Skill Measure: Modified Heidke Skill Score of Categorized Forecasts
Displays and Stratifications
CPC: Summed over all forecasts for each lead and displayed with times series for this lead
Here:
Summed over all forecasts for each lead and all leads displayed together
Stratified further by cold seasons (DJF to FMA) and warm seasons (MAM to NDJ)
Stratified further by strong ENSO years vs. other years
Stratified by region

7. Further Stratification by Strong-ENSO vs Other Years: Temp.

8. Further Stratification by Strong-ENSO vs Other Years: Temp.

9. Displays and Stratifications
CPC Seasonal Forecasts
For 3-equally probable temperature and precipitation classes at 102 Climate Divisions
Made every month from 1995 to present for 0.5-, 1.5-, …, 12.5 month leads
Skill Measure: Modified Heidke Skill Score of Categorized Forecasts
Displays and Stratifications
CPC: Summed over all forecasts for each lead and displayed with times series for this lead
Here:
Summed over all forecasts for each lead and all leads displayed together
Stratified further by cold seasons (DJF to FMA) and warm seasons (MAM to NDJ)
Stratified further by strong ENSO years vs. other years
Stratified by region

11. Stratification by Lead and Regions: Temp.

12. Results Seasonal Temperature:
Useable skill confined to strong ENSO years and mainly at short to medium leads
Otherwise skill is dominantly level with lead (derived from biased climatologies, ie long-term trend)
Best forecasts are for strong-ENSO cold seasons at very short leads
Worst forecasts are for cold seasons at longer leads for strong ENSOs and at very-short leads for other years
Skill is substantially higher than average in the West, and substantially lower in the East
Short-lead forecasts are better now than for the 1960s-80s, ~13 vs ~8 overall, ~20 vs ~13 for the winter

13. Stratification by Lead and Seasons: Precipitation

14. Further Stratification by Strong-ENSO vs Other Years: Precip.

15. Further Stratification by Strong-ENSO vs Other Years: Precip.

17. Stratification by Lead and Regions: Precip.

18. Results Seasonal Precipitation:
Barely useable skill entirely confined to strong ENSO years in short to medium leads
Otherwise skill is either negative or statistically indistinguishable from zero
Best forecasts are for strong-ENSO cold seasons at short leads
Skill is a little higher than average in the South, and a little lower in the North
Short-lead forecasts overall seem to be no better now than for the 1960s-80s

19. Conclusions and Lessons
There are non-random subsets of seasonal forecasts that are skillful enough to be useful
There are many non-random subsets of forecasts that do not have useful skill (should we be issuing them?)
It is critical for this information to be shared with potential users
Skill analyses with different stratifications are highly informative while CPC’s web displays are not
Skills mainly reflect ENSO and trend signals
Temperature forecasts are getting better