Forecasting Performance Measures Performance Measures

Comparing Forecasting Techniques There is more than one approach that can be used to generate a forecast from a set of time series values. –For the stationary Yoho model we used: Last Period 4-Period Moving Average 4-Period Weighted (.4,.3,.2,.1) Moving Average Question: “Which method appears to be the most effective for a particular set of time series values” Answer – The one that comes “the closest” on the average if used to forecast values of the current time series.

Forecast Errors performance measure To try to determine which one of these forecasting methods gives the “best” forecast, we evaluate the “success” of each method using a performance measure. forecast errors  t Each performance measure begins by evaluating the forecast errors  t for each time period t given by:  t = y t - F t What actually happened at time t What was forecasted to happen at time t using the forecast technique Forecast Error At time t

Performance Measures Mean Square Error MSE   t  2  n MSE = Mean Absolute Deviation MAD MAD =  t |  n  Mean Absolute Percent Error MAPE MAPE = n   t |  n y t Largest Absolute Deviation LAD LAD = max |  t |

Which Performance Measure Should Be Used? Choice of the modeler Advantages –MSE gives greater weight to larger deviations (which could result from outliers) –MAD gives less weight to larger deviations –MAPE gives less overall weight to a large deviation if the time series value is large –LAD tells us if all deviations fall below some threshold value Although we illustrate all 4 techniques, in general focus will be on MSE and MAD.

Excel – Performance Measures for Last Period Technique Note: Rows 8-43 are hidden =B3-C3=D3^2 =ABS(D3) =ABS(D3)/B3 Drag cells D3:G3 down to D53:G53 = AVERAGE(E3:E53) = MAX(F3:F53) = AVERAGE(F3:F53) = AVERAGE(G3:G53)

Excel – Performance Measures for 4- Period Moving Average Technique Note: Rows 8-43 are hidden =B6-C6=D6^2 =ABS(D6) =ABS(D6)/B6 Drag cells D6:G6 down to D53:G53 = AVERAGE(E6:E53) = MAX(F6:F53) = AVERAGE(G6:G53) = AVERAGE(F6:F53)

Performance Measures for 4-Period Weighted Moving Average Technique Note: Rows 8-43 are hidden =B6-C6=D6^2 =ABS(D6) =ABS(D6)/B6 Drag cells D6:G6 down to D53:G53 = AVERAGE(E6:E53) = MAX(F6:F53) = AVERAGE(G6:G53) = AVERAGE(F6:F53)

Performance Measures Summary MSE MAD MAPE LAD Last Period Period MA Period WMA Period Moving Average performed best using all performance measures. This is not all ways the case.

Review Performance Measures allow the comparison of the accuracy of various forecasting techniques applied to a particular time series. Performance Measures are all based on deviations of forecasted values from actual values in the time series: Δ t = y t - F t –MSE averages the squared deviations –MAD averages the absolute deviations –MAPE averages the absolute percent errors –LAD is the largest absolute deviation Excel