1. Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Final Results of the NN3 Neural Network Forecasting Competition Sven F. Crone, Konstantinos Nikolopoulos and Michele Hibon

2.  Can NN modelling be automated for business forecasting?  Evaluate progress in NN modelling since M3  Disseminate Explicit knowledge on “best practices” 2005 SAS & IIF Grant RATIONAL OBJECTIVES RESULTS DISCUSSION FURTHER RESEARCH

3. 2005 SAS & IIF Grant RATIONAL

4.  Only 1 evaluation of NN within Forecasting Competitions  Distinct fields of research and participation NN: breakthrough or passing fad? Reid 1969 Santa Fe 1991 BUSINESS FORECASTING COMPETITIONS NN COMPETITIONS Suykens 1998 Reid 1972 Newbold & Granger 1974 Makridakis & Hibon 1979 M-Competition 1982 M2-Competition 1988 M3-Competition 2000 H-Competition, Hibon 2006 EUNITE 2001 ANNEXG 2001 BI Cup 2003 CATS 2005 ISF05 2005 ISF06 ANNEX 2006 WCCI 2006 Only 1 NN entry Balkin & Ord

5.  Most NN competitions = classification ( EUNITE’02, WCCI06 etc.)  Limited evidence on Regression evaluations  Visit http://www.neural-forecasting.com/competition_data.htmhttp://www.neural-forecasting.com/competition_data.htm CI Time Series Competitions Time SeriesData FormatLengthSubmis. SANTA FE 1991 Gershenfeld & Weigend 2 univariate 4 multivariate UV: Laser, UV: Artificial, Sleep, Exchange rate, Astrophysics, Music 1000, 34000, 300000, 100000, 27704, 3808 30+ Black Box 1998 Suykens & Vandewalle 1 univariatePhysics 2000 (1000) 17 EUNITE 20011 multivariateElectrical Load 35040 (31) 56 ANNEXG 2001 Dawson et al. 1 multivariateHydrology 1460 points Hydrology 12 BI Cup 2003 Weber 1 multivariateSugar sales 365 days (14) 10+ CATS 2005, IEEE Lendasse, 1 univariate in 5 parts Artificial 4905pointas (95 points, 5*19) 25 ISF2005 Crone 2 univariateAirline, M3-Competition144, 8516 ANNEXG / ISF2006 Dawson et al., Crone 3 multivariateHydrology 1460 points Hydrology 12 WCCI 2006 Predictive Uncertainty, Gawley 1 univariate 3 multivariate UV: Synthetic, Precipitation, Temperature, SO2 380, 10000, 10000, 21000 9

6.  Conduct competition on industry data  Evaluate different NN methodologies  Can NN forecasting be AUTOMATED on many time series? Reasons? Modelling Decisions Gap between forecasting & NN domains  NN evaluations on different data types  No positive evidence on M-type data Short time series Noisy time series Discouraging research findings  NN cannot forecast seasonal time series  No valid & reliable methodology to model NN  No automation of NN modelling possible

7.  Can NN modelling be automated for business forecasting?  Evaluate progress in NN modelling since M3  Disseminate Explicit knowledge on “best practices” 2005 SAS & IIF Grant OBJECTIVES a)What is the performance (accuracy, robustness & resources) of NN in comparison to established forecasting methods? b)What are the current “best practice” methodologies utilised by researchers to model NN for time series forecasting

8.  Multiple Hypothesis Testing similar to M3-competition Competition Design Multiple empirical Time Series  Complete set of 111 time series  Reduced set of 11 time series  Representative structures  monthly industry data long & short time series Seasonal and non-seasonal series  Scaled observations for anonymity  No domain knowledge  18 steps ahead forecasts Simulated ex ante (out of sample) evaluation Multiple error measures & computational time Testing of conditions under which NN perform well/bad NN3 COMPETITION

9. Competition Design 46 Submissions for the reduced dataset 9 benchmarks 22 submissions for the complete dataset 8 benchmarks Submissions

10. 2005 SAS & IIF Grant Automated AI/CI approaches can very well do the job! (batch forecasting) Balkin’s and Ord approach was not very ‘bad’ after all.. Performance was verified across many metrics (including MASE), parametric + non-parametric Performance was verified with multiple hypothesis: long/short, seasonal/non seasonal, difficult/easy So… WHAT do we know NOW that we did not knew before NN3?

11. 2005 SAS & IIF Grant Time Series Benchmarks are very hard to beat! Forecast Pro, Theta model and Marc Wildi’s Stat benchmark outperform overall all CI/AI approaches For the ‘harder’ part of the NN3 dataset – 25 short+non-seasonal series – CI approaches managed to outperform all other approaches!! Full automation seems to be possible in large scale forecasting tasks + Side results… New Stat benchmarks that perform outstandingly Improvement of established forecasting engines in the last 10 years So… WHAT do we know NOW that we did not knew before NN3?

12. +

13. + +

14. +

20. 2005 SAS & IIF Grant Computational times…. Leaders of the field (Academia + Commercial) Time series features that would necessitate the use of AI/CI approaches Replication in a competition of the M3 volume (NN5…111, tourism competition…1000+) Best practices? Full automation?? and… WHAT we still DO NOT …

21. ? Sven, Kostas & Michele