Machine Learning IV Ensembles CSE 473. © Daniel S. Weld 2 Machine Learning Outline Machine learning: Supervised learning Overfitting Ensembles of classifiers.

Slides:

Advertisements

Similar presentations

Random Forest Predrag Radenković 3237/10

Advertisements

Ensemble Learning Reading: R. Schapire, A brief introduction to boosting.

Machine Learning III Decision Tree Induction

Lectures 17,18 – Boosting and Additive Trees Rice ECE697 Farinaz Koushanfar Fall 2006.

My name is Dustin Boswell and I will be presenting: Ensemble Methods in Machine Learning by Thomas G. Dietterich Oregon State University, Corvallis, Oregon.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

Ensemble Methods An ensemble method constructs a set of base classifiers from the training data Ensemble or Classifier Combination Predict class label.

Model Assessment, Selection and Averaging

Longin Jan Latecki Temple University

Machine Learning CSE 454. © Daniel S. Weld 2 Today’s Outline Brief supervised learning review Evaluation Overfitting Ensembles Learners: The more the.

Machine Learning CSE 454. Administrivia PS1 due next tues 10/13 Project proposals also due then Group meetings with Dan Signup out shortly.

Model Evaluation Metrics for Performance Evaluation

Sparse vs. Ensemble Approaches to Supervised Learning

Ensemble Learning what is an ensemble? why use an ensemble?

Machine Learning II Ensembles & Cotraining CSE 573 Representing Uncertainty.

Ensemble Learning: An Introduction

I.1 ii.2 iii.3 iv.4 1+1=. i.1 ii.2 iii.3 iv.4 1+1=

Three kinds of learning

End of Chapter 8 Neil Weisenfeld March 28, 2005.

Examples of Ensemble Methods

I.1 ii.2 iii.3 iv.4 1+1=. i.1 ii.2 iii.3 iv.4 1+1=

Sparse vs. Ensemble Approaches to Supervised Learning

Ensemble Learning (2), Tree and Forest

For Better Accuracy Eick: Ensemble Learning

Ensembles of Classifiers Evgueni Smirnov

Machine Learning CS 165B Spring 2012

START OF DAY 6 Reading: Chap. 8. Group Project Progress Report.

Issues with Data Mining

Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.

Modified from the slides by Dr. Raymond J. Mooney

ENSEMBLE LEARNING David Kauchak CS451 – Fall 2013.

CS 391L: Machine Learning: Ensembles

LOGO Ensemble Learning Lecturer: Dr. Bo Yuan

Boosting of classifiers Ata Kaban. Motivation & beginnings Suppose we have a learning algorithm that is guaranteed with high probability to be slightly.

1 CS 391L: Machine Learning: Experimental Evaluation Raymond J. Mooney University of Texas at Austin.

Computational Intelligence: Methods and Applications Lecture 36 Meta-learning: committees, sampling and bootstrap. Włodzisław Duch Dept. of Informatics,

Ensemble Methods: Bagging and Boosting

Ensembles. Ensemble Methods l Construct a set of classifiers from training data l Predict class label of previously unseen records by aggregating predictions.

Ensemble Learning Spring 2009 Ben-Gurion University of the Negev.

CLASSIFICATION: Ensemble Methods

ISQS 6347, Data & Text Mining1 Ensemble Methods. ISQS 6347, Data & Text Mining 2 Ensemble Methods Construct a set of classifiers from the training data.

Machine Learning II Decision Tree Induction CSE 573.

Ensemble Methods.  “No free lunch theorem” Wolpert and Macready 1995.

Ensemble Methods in Machine Learning

Classification Ensemble Methods 1

© Daniel S. Weld 2 Course Overview Systems Foundation: Networking, Synchronization & Monitors Datamining Cluster Computing Crawler Architecture Case Studies:

Ensemble Methods Construct a set of classifiers from the training data Predict class label of previously unseen records by aggregating predictions made.

Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.

… Algo 1 Algo 2 Algo 3 Algo N Meta-Learning Algo.

Learning From Observations Inductive Learning Decision Trees Ensembles.

CSE 446: Ensemble Learning Winter 2012 Daniel Weld Slides adapted from Tom Dietterich, Luke Zettlemoyer, Carlos Guestrin, Nick Kushmerick, Padraig Cunningham.

Neural networks (2) Reminder Avoiding overfitting Deep neural network Brief summary of supervised learning methods.

1 Machine Learning Lecture 8: Ensemble Methods Moshe Koppel Slides adapted from Raymond J. Mooney and others.

Machine Learning: Ensemble Methods

Issues in Machine Learning

Ensemble Learning Introduction to Machine Learning and Data Mining, Carla Brodley.

CSE P573 Applications of Artificial Intelligence Decision Trees

A “Holy Grail” of Machine Learing

Combining Base Learners

Data Mining Practical Machine Learning Tools and Techniques

Introduction to Data Mining, 2nd Edition

Machine Learning Ensemble Learning: Voting, Boosting(Adaboost)

CSE 573 Introduction to Artificial Intelligence Decision Trees

Machine Learning CSE 454.

Ensemble learning.

Model Combination.

Ensemble learning Reminder - Bagging of Trees Random Forest

Model generalization Brief summary of methods

Advisor: Dr.vahidipour Zahra salimian Shaghayegh jalali Dec 2017

Presentation transcript:

Machine Learning IV Ensembles CSE 473

© Daniel S. Weld 2 Machine Learning Outline Machine learning: Supervised learning Overfitting Ensembles of classifiers Bagging Cross-validated committees Boosting Stacking

© Daniel S. Weld 3 Ensembles of Classifiers Assume Errors are independent Majority vote Probability that majority is wrong… If individual area is 0.3 Area under curve for  11 wrong is Order of magnitude improvement! Ensemble of 21 classifiers Prob Number of classifiers in error = area under binomial distribution

© Daniel S. Weld 4 Voting

© Daniel S. Weld 5 Constructing Ensembles Generate k sets of training examples For each set Draw m examples randomly (with replacement) From the original set of m examples Each training set corresponds to 63.2% of original (+ duplicates) Now train classifier on each set Bagging

© Daniel S. Weld 6 Ensemble Construction II Partition examples into k disjoint equiv classes Now create k training sets Each set is union of all equiv classes except one So each set has (k-1)/k of the original training data Now train a classifier on each set Cross-validated committees  Train  Test

© Daniel S. Weld 7 Ensemble Construction II Partition examples into k disjoint equiv classes Now create k training sets Each set is union of all equiv classes except one So each set has (k-1)/k of the original training data Now train a classifier on each set Cross-validated committees Test

© Daniel S. Weld 8 Ensemble Construction II Partition examples into k disjoint equiv classes Now create k training sets Each set is union of all equiv classes except one So each set has (k-1)/k of the original training data Now train a classifier on each set Cross-validated committees Test

© Daniel S. Weld 9 Ensemble Creation III Maintain prob distribution over set of training ex Create k sets of training data iteratively: On iteration i Draw m examples randomly (like bagging) But use probability distribution to bias selection Train classifier number i on this training set Test partial ensemble (of i classifiers) on all training exs Modify distribution: increase P of each error ex Create harder and harder learning problems... “Bagging with optimized choice of examples” Boosting

© Daniel S. Weld 10 Ensemble Creation IV Stacking Train several base learners Next train meta-learner Learns when base learners are right / wrong Now meta learner arbitrates Train using cross validated committees Meta-L inputs = base learner predictions Training examples = ‘test set’ from cross validation