1. Ana Galvão, Anthony Garratt and James Mitchell. September 2017
Density Forecasting in Practice: UK output growth and inflation. The WBS Forecasting System
Ana Galvão, Anthony Garratt and James Mitchell.
September 2017

2. WBS Forecasting System - Galvao, Garratt, Mitchell
Outline
(1) What do we do?
(2) How do we do it?
(3) How have we done?
WBS Forecasting System - Galvao, Garratt, Mitchell

3. WBS Forecasting System: What do we do?
Point forecasts, the traditional focus of macroeconomic forecasters, are best interpreted as the central points of ranges of uncertainty.
The Warwick Business School Forecasting System quantifies and communicates these forecast uncertainties by producing probabilistic forecasts.
Addresses the issue of model uncertainty by using combinations of many models.
Calendar-year forecasts of output Growth and inflation are released every quarter at the day of second UK GDP release.
WBS Forecasting System - Galvao, Garratt, Mitchell

4. WBSFS August 2017 Forecasts
These are based on a combination of statistical forecasting models: They are judgement-free calendar-year forecasts!
WBS Forecasting System - Galvao, Garratt, Mitchell

5. WBSFS August 2017 Forecasts
We provide probability forecasts for events: low growth and inflation outside the targeted interval.
The main purpose is to aid decision making.
Year
Real GDP Growth (%, p.a.)
CPI Inflation (%, p.a.)
Prob(growth<0%)
Prob(growth<1%)
Prob(growth<2%)
Prob(letter)
Prob(CPI<1%)
Prob(CPI>3%)
2017
<1% 2%
88%
10%
2018 5%
19%
45%
33%
23%
WBS Forecasting System - Galvao, Garratt, Mitchell

6. WBSFS August 2017 Forecasts
Another objective of the WBS forecasting system is to provide benchmark and judgement-free probability forecasts.
Forecasts for 2018:
Real GDP Growth (%, p.a.)
CPI Inflation (%, p.a.)
Point Forecast
Prob. of a higher outturn
Bank of England
1.68
0.63
2.52
0.39
HMT Panel
1.40
0.71
2.60
0.36
WBS Forecasting System - Galvao, Garratt, Mitchell

7. WBS Forecasting System - Galvao, Garratt, Mitchell
How do we do it? Model Set
Simple Statistical Autoregressive Model:
AR: autoregressive model of order 2 estimated with quarterly growth rates; density forecasts computed by bootstrap (parameter and forecasting uncertainties).
A Mixed Frequency Regression to exploit Monthly Info:
MIDAS: direct regression of quarterly targets on monthly indicators, allowing for one month lead depending on publication delays; Beta weighting function; density forecasts computed by fixed-regressor bootstrap (both uncertainties). We consider 13 indicators so we have 13 single indicator MIDAS models.
WBS Forecasting System - Galvao, Garratt, Mitchell

8. WBS Forecasting System - Galvao, Garratt, Mitchell
Model Set
A “Macroeconomic” BVAR Model:
BVAR_macro: Bayesian VAR model of real GDP and CPI quarterly macro series (Consumption, Investment, Hours, Real Wages, Bank Rate). Variables in levels, p=4; Normal/Wishart to obtain Density Forecasts.
“Leading Indicator” BVAR Models:
BVAR_medium: Bayesian VAR model of real GDP and CPI indicators (as in the MIDAS models). Normal/Wishart to obtain Density Forecasts. 8 different specifications: p=1,…,4 x Levels/Differences.
A Time-Varying BVAR with Stochastic Volatility:
TVP_VAR: A model of output growth, inflation and the short- rate. p=1. Gibbs + MH step draws to obtain density forecasts [H. Mumtaz code].
WBS Forecasting System - Galvao, Garratt, Mitchell

9. WBS Forecasting System - Galvao, Garratt, Mitchell
Our Information Set
Name
Publication Delay
Output Growth
56 days (2nd)
Inflation
14 days
Consumption
56 days
Investment
Real Wages
75 days
Hours
Bank rate
no delay
Industrial Prod.
45 days
Business Confidence
Employment
Unemployment
Consumer Confidence
House Prices
Stock Prices
Exchange Rates
10 days
House Prices _2
Short rate
9 days
Yield Spread
Retail Prices
Oil Prices
29 days
WBS Forecasting System - Galvao, Garratt, Mitchell

10. WBS Forecasting System - Galvao, Garratt, Mitchell
Why combine?
Point forecasts are more robust to instability/structural breaks since models’ relative forecasting performance may change over time.
It might be easier to find density forecasts that approximate well the true data density.
Equal weighting might be an inadequate combination method: it does not down weight bad forecasters … Weighting scheme based on past performance may perform better than equal weighting if forecasters have very different performance.
Evaluation with quarterly fixed horizon forecast suggested that log pool performs better than the linear pool.
WBS Forecasting System - Galvao, Garratt, Mitchell

11. WBS Forecasting System - Galvao, Garratt, Mitchell
Combination Method
Total number of models/experts: 24.
Logarithm opinion pool:
𝑑𝑒𝑛𝑠 𝑡+ℎ|𝑡 = 𝑖=1 𝑚 ( 𝑑𝑒𝑛𝑠 𝑡+ℎ|𝑡 𝑖 ) 𝑤(𝑖) , where rescaling is required.
Selecting Model’s (Expert’s) weights:
Proportional to the relative forecasting performance up to t measured by the logscore. This uses a rolling window of 12 quarterly observations. Weights estimated to fixed horizon quarterly forecasts and using the second monthly release of GDP as the actual value (real-time!)
WBS Forecasting System - Galvao, Garratt, Mitchell

12. Model weights: time-varying, specific to period and variable forecast
2017q2 (h=3,7)
2017 –output
2018 -output
inflation
inflation ar
0.057
0.042
0.131
0.052
bvarmacro
0.054
0.047
0.067
0.051
tvpvar
0.020
0.025
0.028
0.024 ip
0.041
0.000
0.038
bus conf
0.044
0.001
emp
0.061
0.056
0.075
unem
0.017
0.045
cons conf
0.027
0.031
house pr
0.037
stock pr
0.046
exc rates
0.053
0.043
house pr 2
0.064
0.032
short rate
0.018
0.030
0.040
yield sp
0.039
0.036
retail prices
oil prices
L-bvarmed_p1
0.026
L-bvarmed_p2
L-bvarmed_p3
0.050
0.023
0.035
L-bvarmed_p4
0.014
D-bvarmed_p1
0.217
D-bvarmed_p2
0.162
D-bvarmed_p3
0.117
0.048
D-bvarmed_p4
0.109
2017q3 (h=2, 6)
2017 -output
2018 -output
inflation
inflation ar
0.0542
0.0437
0.7325
0.0593
bvarmacro
0.0427
0.0468
0.0011
0.0552
tvpvar
0.022
0.0264
0.0159
0.0167 ip
0.0576
0.0477
0.0362
bus conf
0.0461
0.0454
0.04
emp
0.0596
0.0567
0.0556
unem
0.0473
0.051
0.0448
cons conf
0.0307
0.0155
0.028
house pr
0.05
0.0425
0.0428
stock pr
0.053
0.0353
exc rates
0.0503
0.0475
0.0267
house pr 2
0.0621
0.056
0.0326
short rate
0.0348
0.0242
0.0424
yield sp
0.0458
0.0422
0.0393
retail prices
0.039
0.0455
0.0401
oil prices
0.0299
0.0132
0.0467
L-bvarmed_p1
0.0201
0.0511
0.1358
0.0397
L-bvarmed_p2
0.0313
0.0004
0.0333
L-bvarmed_p3
0.0303
0.0482
0.0311
L-bvarmed_p4
0.029
0.048
0.0271
D-bvarmed_p1
0.047
0.0391
0.0569
0.0555
D-bvarmed_p2
0.0412
0.0379
0.032
0.0572
D-bvarmed_p3
0.038
0.0559
D-bvarmed_p4
0.0381
0.0389
0.0243
0.0586
WBS Forecasting System - Galvao, Garratt, Mitchell

13. WBS Forecasting System - Galvao, Garratt, Mitchell
How have we done?
Formal evaluation difficult because
Just 12 forecasting rounds, three of which (for 2017) with respect unknown future outcomes
Fixed event nature of forecast events – therefore effectively have just three/two observations per event
More formal analysis undertaken when choosing models.
Therefore here we tabulate and plot some probabilities ties and outcomes, as well as the history of our benchmarking exercise
WBS Forecasting System - Galvao, Garratt, Mitchell

14. WBS Forecasting System - Galvao, Garratt, Mitchell
Table 1: Output Growth, Probabilities and Outturns
Probability of current year growth
Probability of next year growth
<0%
<1%
<2%
Obs.
2014q1
0.00
3.07
0.03
0.07
0.22
2.19
2015q1
0.01
0.05
0.24
0.06
0.14
0.29
1.81
2015q2
0.27
0.13
2015q3
0.02
0.08
0.25
2015q4
0.28
2016q1
0.10
0.45
2016q2
0.43
2016q3
2016q4
0.23
WBS Forecasting System - Galvao, Garratt, Mitchell

15. WBS Forecasting System - Galvao, Garratt, Mitchell
Table 2: Inflation, Probabilities and Outturns
Probability of current year inflation
Probability of next year inflation
<0%
<1%
<2%
Obs.
2014q1
0.00
1.00
1.41
0.12
0.41
0.76
0.04
2015q1
0.44
0.87
0.31
0.58
0.82
0.65
2015q2
0.84
0.99
0.50
2015q3
0.29
2015q4
0.19
0.74
0.96
2016q1
2016q2
2016q3
2016q4
WBS Forecasting System - Galvao, Garratt, Mitchell

16. WBS Forecasting System - Galvao, Garratt, Mitchell
Table 3: Probability of NOT being in inflation target range of 1-3% i.e. writing a letter
Probability in current year
Probability next year
Prob.
Obs.
2014q1
0.00
Hit
1.41
0.46
No hit
0.04
2015q1
0.87
0.63
0.65
2015q2
0.99
0.52
2015q3
1.00
2015q4
0.73
2016q1
2016q2
2016q3
2016q4
WBS Forecasting System - Galvao, Garratt, Mitchell

17. WBS Forecasting System - Galvao, Garratt, Mitchell

18. WBS Forecasting System - Galvao, Garratt, Mitchell

19. WBS Forecasting System - Galvao, Garratt, Mitchell

20. WBS Forecasting System - Galvao, Garratt, Mitchell

21. WBS Forecasting System Future
Provide quarterly releases of calendar-year probabilistic forecasts based on state-of-art forecasting models.
Forecasts are published at: casting/
Update model set/combination weights based on comprehensive research on density forecasting combination.
Keep forecasts judgment free so we can evaluate the extension of judgment in alternative published forecasts.
WBS Forecasting System - Galvao, Garratt, Mitchell

22. APPENDIX: Evaluating WBSFS Forecasts
We use a long out-of-sample evaluation period. Forecasts are computed from 1999Q1 to 2012Q3.
Real-time exercise: only data actually available at the time forecast is computed (ONS monthly real-time dataset of real GDP; and for other macro variables) and takes into account publication delays of series not subject to revisions.
Quarterly forecast horizons from h=1 (current quarter forecast) up to h=8 (two-year ahead forecast).
[2015 forecast computed in Feb 2015 requires forecasts for h=1 up to h=4; if computed in Nov 2015, only h=1 forecast is required since previous quarters are available].
WBS Forecasting System - Galvao, Garratt, Mitchell

23. Evaluating WBSFS Forecasts
Measure of point forecast accuracy: root mean squared forecast error (RMSFE).
Measures of density forecasting accuracy: logscore and continuous ranking probability score (CRPS).
The measures above allow us to test for statistical significant differences in forecasting performance.
Measures of calibration of density forecasts. Being well- calibrated means approximating well the true data density. We use tests based on probability integral transforms (PITs).
WBS Forecasting System - Galvao, Garratt, Mitchell

24. Comparing Accuracy with BofE
Output Growth ( N=55)
Point Forecast
Density Forecast – CRPS score
Density Forecast – Log Score 1 2 4 8
BofE
0.526
0.80
1.588
2.364
0.307
0.439
0.822
1.17
4.493
4.704
5.823
6.474 AR
1.10
1.34
1.29
0.97
1.02
1.27
1.23
0.96
1.05
1.00
0.89
BVARmed
0.82
1.06
1.01
1.03
0.98
0.94
Comb; m=2;
1.07
1.12
0.93
1.11
Comb; m=5
1.19
1.32
1.18
1.04
0.99
Comb; m=24;log
0.88
1.24
1.21
1.15
1.14
1.09
1.08
Inflation(N=35; from 2004).
Point Forecast
Density Forecast – CRPS score
Density Forecast – Log Score 1 2 4 8
BofE
0.184
0.483
1.088
1.512
0.130
0.283
0.625
0.845
3.55
4.214
5.127
5.449 AR
2.11
1.54
1.16
0.60
1.65
1.44
1.12
0.68
1.01
0.93
MIDAS
1.75
1.30
1.18
0.86
1.37
1.20
0.84
1.06
1.05
0.95
BVARmed
2.34
1.84
1.64
1.48
1.74
1.66
1.55
1.34
1.13
1.25
1.07
1.03
Comb; m=2
2.19
1.35
0.98
1.50
1.26
0.90
0.96
Comb; m=5
2.15
1.77
1.47
1.59
1.52
1.14
1.10
1.08
1.02
Comb; m=24;log
1.19
0.87
1.21
1.17
WBS Forecasting System - Galvao, Garratt, Mitchell

25. Calibration of Density Forecasts
Output Growth ( N=55)
Mean PITs: 0.5
(Berkowitz Test)
Std Deviation PITs:
(Knuppel Test) 1 2 4 8
BofE
0.41
0.40
0.35
0.34
0.29
0.27
0.26
0.24 AR
0.45
0.37
0.21
BVARmed
0.49
0.51
0.28
Comb_2
0.47
0.48
0.46
0.44
Comb_5
0.43
0.25
0.22
Comb_24_log
0.39
Inflation(N=35; from 2004).
Mean PITs: 0.5
(Berkowitz Test)
Std Deviation PITs:
(Knuppel Test) 1 2 4 8
BofE
0.34
0.37
0.45
0.49
0.30
0.36
0.41 AR
0.48
0.44
0.29
0.28
0.26
0.19
MIDAS
0.52
0.51
0.56
0.23
BVARmed
0.50
0.24
Comb_2
0.27
0.21
Comb_5
0.55
0.57
0.60
0.61
0.22
Comb_24_log
0.53
WBS Forecasting System - Galvao, Garratt, Mitchell

26. Main Evaluation Messages
The BofE Inflation Report Forecasts of Output Growth and Inflation perform well if compared with statistical models in particularly when predicting current year inflation.
Their predictive density however are not good approximations of the true data density.
By combining a set of statistical models, we can obtain well- calibrated densities for current year forecasts of output growth and inflation. [we can also to get well-calibrated two-years ahead inflation forecast].
Regular releases of well-calibrated predictive densities of output growth and inflation, which help decision making, is one of the main purposes of the WBS Forecasting System.
WBS Forecasting System - Galvao, Garratt, Mitchell