1. 1 Ensembles of Nearest Neighbor Forecasts Dragomir Yankov, Eamonn Keogh Dept. of Computer Science & Eng. University of California Riverside Dennis DeCoste Yahoo! Research

2. 2 Outline Problem formulation NN forecasting framework Stability of the forecasts Ensembles of NN forecasts Experimental evaluation

3. 3 Problem formulation Predict the number of impressions to be observed for a specific website Data specifics – many patterns present in the data

4. 4 Forecasting framework – overview

5. 5 Forecasting framework – formalization Formalization –Direct forecasts: Given: a query, its k nearest neighbors Estimate: the query continuation –Other approaches: iterative forecasts, mutually validating forecasts

6. 6 Forecasting framework – components Similarity measure –Standardized Euclidean distance: where Prediction accuracy –Prediction root mean square error: Weighting function – uniform weights

7. 7 Stability of the forecasts Stability with respect to the training data –NN is stable in the case of classification and majority voting (Breiman ’96) –Here – extrapolation plus regression. Changing one neighbor can change the forecast significantly Stability with respect to the input parameters –Parameters: k, weights of different neighbors, query length, prediction horizon –Different combinations lead to different forecasts

8. 8 Ensembles of NN forecasts Main idea: rather than tuning up the best parameters for the entire dataset, for each query select the model that will predict it best Issues –What base models to use –How to select among them

9. 9 Ensembles of NN forecasts Base models to use –We focus on pairs of NN learners, in which the base models differ in the number of neighbors used –The optimal single predictors and the suitable ensembles are determined on a validation set using an oracle kRMSE (k-NN)(k1, k2)RMSE (Ens) 12.0447(1, 20)1.5829 21.9504(2, 40)1.5996 61.8321(6, 1)1.6305 101.8387(10, 1)1.5953 1002.9608(100, 1)1.6095 Optimal Single Predictor Optimal Ensemble (Using Oracle)

10. 10 Ensembles of NN forecasts Selecting among the base models: –Learn a classifier to select the more suitable model for individual queries (SVM with Gaussian kernel) Note: The classifier does not need to be perfect. It is important to identify the “bad” cases for each base learner

11. 11 Ensembles of NN forecasts Selecting among the base models: –Extracted features: Statistics from the query and its nearest neighbors: Mean, Median, Variance, Amplitude Statistics from the models’ forecasts: Mean, Median, Variance, Amplitude Distances between the forecasts of the individual neighbors Performance of the models on the query’s nearest neighbors Step-back forecasts (good for short horizons)

12. 12 Experimental evaluation Website impressions

13. 13 Experimental evaluation Website impressions –Computing the optimal single predictors –Comparison with the accuracy of the ensemble approach HorizonPredictorTest RMSEStd h = 3010-NN (optimal k)1.1230.644 Ens = {10-NN,1-NN}1.0210.452 h = 608-NN (optimal k)1.5490.862 Ens = {10-NN,1-NN}1.4120.685 h = 1006-NN (optimal k)1.8671.183 Ens = {10-NN,1-NN}1.6880.961

14. 14 Experimental evaluation Website impressions

15. 15 Experimental evaluation Bias-Variance improvement –We compute the bias 2 and variance terms in the error decomposition for h=100 steps ahead –The statistics are recorded over 50 random subsamples from the original training set PredictorBias 2 Variance 6-NN (optimal k)5.0420.638 Ens = {10-NN,1-NN}3.7210.204

16. 16 Conclusions and future directions The proposed technique improves significantly the prediction accuracy of the single NN forecasting models It outlines a principled solution to the bias-variance problem of the NN forecasts It is a data specific rather than a generic approach Combining more models and varying other parameters would require selecting different features Thank you!