Presentation on theme: "When Efficient Model Averaging Out-Perform Bagging and Boosting Ian Davidson, SUNY Albany Wei Fan, IBM T.J.Watson."— Presentation transcript:
When Efficient Model Averaging Out-Perform Bagging and Boosting Ian Davidson, SUNY Albany Wei Fan, IBM T.J.Watson
Ensemble Techniques Techniques such as boosting and bagging are methods of combining models. Used extensively in ML and DM seems to work well in a large variety of situations. But model averaging is the correct Bayesian method of using multiple models. Does model averaging have a place in ML and DM?
What is Model Averaging? Posterior weighting Class Probability Integration Over Model Space Averaging of class probabilities weighted by posterior Removes model uncertainty by averaging Prohibitive for large model spaces such as decision trees
Efficient Model Averaging: PBMA and Random DT PBMA (Davidson 04): parametric bootstrap model averaging –Use parametric model to generate multiple bootstraps computed from a single training set. Random Decision Tree (Fan et al 03) –Construct each trees structure randomly Categorical feature used once in a decision path Random threshold for continuous features. –Leaf node statistics estimated from data. –Average probability of multiple trees.
Our Empirical Study Idea: When model uncertainty occurs, model averaging should perform well Four specific but common situations when factoring in model uncertainty is beneficial –Class label noise –Many label problem –Sample selection bias –Small data sets
Scope of This Paper Identifies conditions where model averaging should outperform bagging and boosting. Empirically verifies these claims. Other questions: –Why does bagging and boosting perform badly in these conditions?