1. A STATISTICAL COMPARISON OF AMPS 10-KM AND 3.3-KM DOMAINS Michael G. Duda, Kevin W. Manning, and Jordan G. Powers Mesoscale and Microscale Meteorology Division, NCAR AMPS Users’ Workshop 2004 June 8-10, 2004

2. Introduction Purpose: –Demonstrate the usefulness of statistical significance testing in comparing biases of two domains –Determine where biases at McMurdo Station are significantly different in the 3.3-km and 10-km AMPS domains –Examine a 7 day period beginning 12Z Nov. 27, 2003 when McMurdo Station was affected by a snowstorm Methodology: –Use hypothesis testing to identify statistically significant differences in mean bias –Consider only differences that are statistically significant

3. Domain Configuration Compaq OSF/AlphaLinux/Xeon (SPAWAR machine)

4. Forecast Analysis Times

5. Why Consider Statistical Significance? Mean bias curves do not indicate the variance in the biases Some differences between curves are not as relevant

6. Hypothesis Testing Consider biases to be from a hypothetical population (assumed to be normally distributed) Let d = x 3.3 – x 10 –x 3.3 and x 10 are biases in 3.3-km and 10-km domains at a given time Perform one-sample Student’s t test H 0 :  d =0 Reject H 0 with 95% confidence if t  t  Test statistic:

7. Hypothesis Testing Example Circled pressure levels will be examined in the next two slides

8. Example: 150 hPa Temperature differences between curves For this data we can reject the null hypothesis at the 5 percent level This means we reject the hypothesis that the means of the 3.3-km and 10- km bias populations are the same

9. Example: 850 hPa Temperature differences between curves For this data we cannot reject the null hypothesis at the 5 percent level This means we cannot reject the hypothesis that the 3.3-km and 10-km bias populations have the same mean

10. Comparison Results: Temperature Statistically significant differences –Surface: 3.3-km grid has warm bias while 10- km grid has a cool bias at hours 24, 36 –925 hPa: 3.3-km grid has warm bias while 10- km grid has a cool bias at hours 24, 36 –300 hPa: 3.3-km grid has larger warm bias than 10-km grid No statistically significant differences –At hours 24 and 36, no significant differences in MAE at any level

11. 24hr Temperature (Mean Bias)

12. 36hr Temperature (Mean Bias)

13. 24hr Temperature (MAE)

14. Comparison Results: Wind U-Component Statistically significant differences –Surface: 3.3-km grid has lower positive bias than 10-km grid at forecast hours 12, 24, 36 –850 hPa: 3.3-km grid has larger negative bias at forecast hours 12, 24, 36 –500 hPa: 3.3-km grid has smaller bias, but MAEs of both grids are similarly large Differences at other levels are not statistically significant

15. 24hr Wind U-Component (Mean Bias)

16. 36hr Wind U-Component (Mean Bias)

17. 24hr Wind U-Component (MAE)

18. Example: Surface Temperature 35 hr forecast valid 23Z Dec 01, 2003 10-km domain 3.3-km domain

19. Summary Use a Student’s t test (at 5 percent level) to perform statistical significance testing on difference between 3.3-km and 10-km biases Identify statistically significant differences on model bias v. pressure plots for McMurdo Consider only statistically significant differences between mean biases to improve objectivity –Apparently large differences in mean bias may be statistically insignificant and misleading

20. Questions?

21. Hypothesis Testing Example * * * Biases at these pressure levels will be examined in the following slides

22. Example: 400 hPa Wind V-Component differences between curves For this data we do not reject the null hypothesis at the 95 percent level

23. Example: 925 hPa Wind V-Component differences between curves For this data we do reject the null hypothesis at the 95 percent level