Model Selection and Assessment Using Cross-indexing Juha Reunanen ABB, Web Imaging Systems, Finland.

Slides:

Advertisements

Similar presentations

Date: 2013/1/17 Author: Yang Liu, Ruihua Song, Yu Chen, Jian-Yun Nie and Ji-Rong Wen Source: SIGIR12 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Adaptive.

Advertisements

Machine Learning: Intro and Supervised Classification

Lectures 17,18 – Boosting and Additive Trees Rice ECE697 Farinaz Koushanfar Fall 2006.

CPSC 502, Lecture 15Slide 1 Introduction to Artificial Intelligence (AI) Computer Science cpsc502, Lecture 15 Nov, 1, 2011 Slide credit: C. Conati, S.

Learning Algorithm Evaluation

Model Assessment and Selection

Model Assessment, Selection and Averaging

Model assessment and cross-validation - overview

CMPUT 466/551 Principal Source: CMU

Autocorrelation and Linkage Cause Bias in Evaluation of Relational Learners David Jensen and Jennifer Neville.

The loss function, the normal equation,

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

HMM-BASED PATTERN DETECTION. Outline  Markov Process  Hidden Markov Models Elements Basic Problems Evaluation Optimization Training Implementation 2-D.

Ensemble Learning: An Introduction

Three kinds of learning

End of Chapter 8 Neil Weisenfeld March 28, 2005.

Model Selection. Agenda Myung, Pitt, & Kim Olsson, Wennerholm, & Lyxzen.

Ensemble Learning (2), Tree and Forest

For Better Accuracy Eick: Ensemble Learning

A Study of Computational and Human Strategies in Revelation Games 1 Noam Peled, 2 Kobi Gal, 1 Sarit Kraus 1 Bar-Ilan university, Israel. 2 Ben-Gurion university,

Model Assessment and Selection Florian Markowetz & Rainer Spang Courses in Practical DNA Microarray Analysis.

Dependency networks Sushmita Roy BMI/CS 576 Nov 26 th, 2013.

RESULTS OF THE WCCI 2006 PERFORMANCE PREDICTION CHALLENGE Isabelle Guyon Amir Reza Saffari Azar Alamdari Gideon Dror.

RESULTS OF THE NIPS 2006 MODEL SELECTION GAME Isabelle Guyon, Amir Saffari, Gideon Dror, Gavin Cawley, Olivier Guyon, and many other volunteers, see

Efficient Model Selection for Support Vector Machines

沈致远. Test error(generalization error): the expected prediction error over an independent test sample Training error: the average loss over the training.

Bias and Variance Two ways to measure the match of alignment of the learning algorithm to the classification problem involve the bias and variance. Bias.

Stacked Generalization an overview of the paper by David H. Wolpert Jim Ries CECS 477 : Neural Networks February 2, 2000.

NFL Play Predictions Will Burton, NCSU Industrial Engineering 2015

ECE 8443 – Pattern Recognition Objectives: Bagging and Boosting Cross-Validation ML and Bayesian Model Comparison Combining Classifiers Resources: MN:

Data Mining - Volinsky Columbia University 1 Topic 10 - Ensemble Methods.

Introduction to Machine Learning Supervised Learning 姓名 : 李政軒.

Ensemble Methods: Bagging and Boosting

Ensembles. Ensemble Methods l Construct a set of classifiers from training data l Predict class label of previously unseen records by aggregating predictions.

CLASSIFICATION: Ensemble Methods

Manu Chandran. Outline Background and motivation Over view of techniques Cross validation Bootstrap method Setting up the problem Comparing AIC,BIC,Crossvalidation,Bootstrap.

PSMS for Neural Networks on the Agnostic vs Prior Knowledge Challenge Hugo Jair Escalante, Manuel Montes and Enrique Sucar Computer Science Department.

ISQS 6347, Data & Text Mining1 Ensemble Methods. ISQS 6347, Data & Text Mining 2 Ensemble Methods Construct a set of classifiers from the training data.

Presentation Title Department of Computer Science A More Principled Approach to Machine Learning Michael R. Smith Brigham Young University Department of.

ISCG8025 Machine Learning for Intelligent Data and Information Processing Week 3 Practical Notes Application Advice *Courtesy of Associate Professor Andrew.

INTRODUCTION TO Machine Learning 3rd Edition

Slides for “Data Mining” by I. H. Witten and E. Frank.

Concept learning, Regression Adapted from slides from Alpaydin’s book and slides by Professor Doina Precup, Mcgill University.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

Guest lecture: Feature Selection Alan Qi Dec 2, 2004.

Bias and Variance of the Estimator PRML 3.2 Ethem Chp. 4.

Ensemble Methods Construct a set of classifiers from the training data Predict class label of previously unseen records by aggregating predictions made.

Validation methods.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Genetic Algorithms (in 1 Slide) l GA: based on an analogy to biological evolution l Each.

Evolving RBF Networks via GP for Estimating Fitness Values using Surrogate Models Ahmed Kattan Edgar Galvan.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Bagging and Boosting Cross-Validation ML.

Evaluating Classifiers. Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website)

Tree and Forest Classification and Regression Tree Bagging of trees Boosting trees Random Forest.

Overfitting, Bias/Variance tradeoff. 2 Content of the presentation Bias and variance definitions Parameters that influence bias and variance Bias and.

Feasibility of Using Machine Learning Algorithms to Determine Future Price Points of Stocks By: Alexander Dumont.

1 Machine Learning Lecture 8: Ensemble Methods Moshe Koppel Slides adapted from Raymond J. Mooney and others.

Ch 1. Introduction Pattern Recognition and Machine Learning, C. M. Bishop, Updated by J.-H. Eom (2 nd round revision) Summarized by K.-I.

Evaluating Classifiers

An Empirical Comparison of Supervised Learning Algorithms

Boosting and Additive Trees (2)

Demographics and Weblog Hackathon – Case Study

Bias and Variance of the Estimator

CSE P573 Applications of Artificial Intelligence Decision Trees

CSE 573 Introduction to Artificial Intelligence Decision Trees

Chap. 7 Regularization for Deep Learning (7.8~7.12 )

Ensemble learning Reminder - Bagging of Trees Random Forest

MILESTONE RESULTS Mar. 1st, 2007

Shih-Yang Su Virginia Tech

Introduction to Machine learning

Presentation transcript:

Model Selection and Assessment Using Cross-indexing Juha Reunanen ABB, Web Imaging Systems, Finland

Model Selection Using Cross-Validation  Choose a search algorithm – for example: hill-climbing, grid search, genetic algorithm  Evaluate the models using cross-validation  Select the model that gives the best CV score

Multiple-Comparison Procedure ( D. D. Jensen and P. R. Cohen: Multiple Comparisons in Induction Algorithms, Machine Learning, volume 38, pages 309–338, 2000)  Example: Choosing an investment advisor  Criterion: Predict stock market change (+/–) correctly for 11 out of 14 days  You evaluate 10 candidates  Your friend evaluates 30 candidates  If everyone is just guessing, your probability of accepting is 0.253, your friend’s 0.583

The Problem  Overfitting on the first level of inference: Increasing model complexity may decrease the training error while the test error goes up  Overfitting on the second level of inference: Making the search more intense may decrease the CV error estimate, even if the test error would actually go up

Overfitting Visualized Model Complexity, or Number of Models Evaluated

Solutions  First level of inference: Regularization – penalize complex models Model selection – welcome to the second level...  Second level of inference: Regularization! (G. C. Cawley and N. L. C. Talbot: Preventing overfitting during model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages , 2007) Another layer of (cross-)validation...

Another Layer of Validation  A lot of variance: the estimate related to the winner gets biased (in the MCP sense)  Cross-validation makes it smoother, but does not remove the problem

The Cross-indexing Trick  Assume an outer loop of cross-validation using five folds  Use (for example) three folds to determine the best depth, and the rest two to assess it This essentially removes the multiple-comparison effect  Revolve, and average (or, create an ensemble)  Previously shown to work in feature selection (Juha Reunanen: Less Biased Measurement of Feature Selection Benefits, SLSFS 2005, LNCS 3940, pages 198–208, 2006)

Competition Entries  Stochastic search guided by cross-validation  Several candidate models (and corresponding search processes running pseudo-parallel): Prepro+naiveBayes, PCA+kernelRidge, GS+kernelRidge, Prepro+linearSVC, Prepro+nonlinearSVC, Relief+neuralNet, RF, and Boosting (with neuralNet, SVC and kernelRidge)  Final selection and assessment using the cross-indexing criterion

Milestone Results Yellow: CLOP model. CLOP prize winner: Juha Reunanen (both ave. rank and ave. BER). Best ave. BER held by Reference (Gavin Cawley) with “the bad”. Agnostic learning ranks as of December 1 st, 2006

Models Selected

Conclusions  Because of multiple-comparison procedures (MCPs) on the different levels of inference, validation is often used to estimate final performance  On the second level, the cross-indexing trick may give estimates that are less biased (when comparing to straightforward outer-loop CV)