1. Econ 427 lecture 24 slides
Forecast Evaluation
Byron Gangnes

2. Review: Linear Projection
Consider the h-step ahead forecast for the following general covariance stationary model:
Wold (MA) representation for the series of interest is:
The h-step-ahead linear least squares forecast is obtained by writing out the series for period T+h,
Byron Gangnes

3. Linear Projection
and then projecting on the info set available today (time T). (See Ch. 9) This gives:
The h-step-ahead forecast error is then:
This is an MA(h-1)
And the forecast error variance is:
Byron Gangnes

4. Optimal Forecasts Four key properties follow:
Optimal forecasts are unbiased.
Optimal forecasts have 1-step-ahead forecasts errors that are white noise
Optimal forecasts have h-step-ahead forecasts that are at most MA(h-1)
We saw above that optimal forecasts will have MA(h-1) errors.
Optimal forecasts have h-step-ahead errors with variances that are non-decreasing in h and that converge to the unconditional variance of the process (series).
Byron Gangnes

5. Test of forecast bias
To test for forecast bias, regress the forecast errors on a constant:
But we need to be sure that residuals from this regression are white noise. So we select an ARMA model to do so and then estimate:
See the discussion in the book of how we can test the other characteristics of optimal forecasts.
Byron Gangnes

6. Mincer-Zarnowitz regression
A key overarching property: forecast errors should be unforecastable on the basis of info avail at the time the forecast is made.
Testing this with a Mincer-Zarnowitz regression:
Then use an F-test to test whether β0=0 and β1=1.
Note that in this case , yt+h = yt+h,t+ ut
they only differ by unforecastable error.
Byron Gangnes

7. Statistical tests of forecast accuracy
At the beginning of the term we looked at various loss functions such as MSE, RMSE, etc.
We may want to test whether the “expected loss differential” between two forecasts a and b is significantly diff from zero:
In the book (pp ) they show a nonparametric test you can do to test whether one forecast is statistically smaller loss than the other.
Byron Gangnes

8. Statistical tests of forecast accuracy
How can we test whether the difference d is significant using and OLS regression?
Regress d on a constant and look at the t-stat on the constant term, β
Note that here you have to model the error (which won’t nec. be white noise—why not?—in order to get an unbiased est. of c term.
Byron Gangnes

9. Forecasting encompassing and forecast combination
It may be the case that one forecast encompasses all of the relevant info in another candidate forecast. In that case your forecast cannot be improved by using the second forecast.
But it is frequently the case that alternative forecasts do not encompass each other and so combining them can yield improved forecast performance.
Byron Gangnes

10. Forecast Combination How can we do this?
Regression-based method of forecast combination. Run the regression:
The regression equation basically lets the data tell us what the optimal weights are to put on the two forecasts.
Normally you will want to allow for ARMA(p,q) error terms since we know that even optimal forecasts can have MA(h-1) errors and non-optimal forecasts could have other dynamics.
Byron Gangnes

11. Forecast Combination
These methods can be applied to any forecasts. They don’t have to be formal model-based forecasts.
See the application at the end of the chapter. Go through it carefully to understand the procedure they follow to compare and combine the two forecasts and the resulting gains. How big are they?
We’ll look at an example next time.
Byron Gangnes