Decision Tree Pruning Methods Validation set – withhold a subset (~1/3) of training data to use for pruning –Note: you should randomize the order of training.

Slides:

Advertisements

Similar presentations

COMP3740 CR32: Knowledge Management and Adaptive Systems

Advertisements

Rule-Based Classifiers. Rule-Based Classifier Classify records by using a collection of “if…then…” rules Rule: (Condition)  y –where Condition is a conjunctions.

1er. Escuela Red ProTIC - Tandil, de Abril, Decision Tree Learning 3.1 Introduction –Method for approximation of discrete-valued target functions.

Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 1.

ICS320-Foundations of Adaptive and Learning Systems

Classification Techniques: Decision Tree Learning

Chapter 7 – Classification and Regression Trees

Chapter 7 – Classification and Regression Trees

Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.

Decision Trees Jeff Storey. Overview What is a Decision Tree Sample Decision Trees How to Construct a Decision Tree Problems with Decision Trees Decision.

Decision Trees Instructor: Qiang Yang Hong Kong University of Science and Technology Thanks: Eibe Frank and Jiawei Han.

Decision Tree Rong Jin. Determine Milage Per Gallon.

ID3 Algorithm Abbas Rizvi CS157 B Spring What is the ID3 algorithm? ID3 stands for Iterative Dichotomiser 3 Algorithm used to generate a decision.

Part 7.3 Decision Trees Decision tree representation ID3 learning algorithm Entropy, information gain Overfitting.

Lazy Associative Classification By Adriano Veloso,Wagner Meira Jr., Mohammad J. Zaki Presented by: Fariba Mahdavifard Department of Computing Science University.

CS 590M Fall 2001: Security Issues in Data Mining Lecture 4: ID3.

Decision Tree Learning Learning Decision Trees (Mitchell 1997, Russell & Norvig 2003) –Decision tree induction is a simple but powerful learning paradigm.

Decision Trees and Decision Tree Learning Philipp Kärger

Decision Trees Decision tree representation Top Down Construction

Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.

Example of a Decision Tree categorical continuous class Splitting Attributes Refund Yes No NO MarSt Single, Divorced Married TaxInc NO < 80K > 80K.

Ch 3. Decision Tree Learning

Classification.

Fall 2004 TDIDT Learning CS478 - Machine Learning.

Machine Learning Chapter 3. Decision Tree Learning

Decision Trees Advanced Statistical Methods in NLP Ling572 January 10, 2012.

Learning what questions to ask. 8/29/03Decision Trees2  Job is to build a tree that represents a series of questions that the classifier will ask of.

Machine Learning Decision Tree.

Mohammad Ali Keyvanrad

Decision tree learning Maria Simi, 2010/2011 Inductive inference with decision trees  Decision Trees is one of the most widely used and practical methods.

Machine Learning Lecture 10 Decision Tree Learning 1.

CpSc 810: Machine Learning Decision Tree Learning.

Decision Trees DefinitionDefinition MechanismMechanism Splitting FunctionSplitting Function Issues in Decision-Tree LearningIssues in Decision-Tree Learning.

Decision Tree Learning

Artificial Intelligence Project #3 : Analysis of Decision Tree Learning Using WEKA May 23, 2006.

For Monday No new reading Homework: –Chapter 18, exercises 3 and 4.

CS 8751 ML & KDDDecision Trees1 Decision tree representation ID3 learning algorithm Entropy, Information gain Overfitting.

1 Universidad de Buenos Aires Maestría en Data Mining y Knowledge Discovery Aprendizaje Automático 5-Inducción de árboles de decisión (2/2) Eduardo Poggi.

CS 5751 Machine Learning Chapter 3 Decision Tree Learning1 Decision Trees Decision tree representation ID3 learning algorithm Entropy, Information gain.

1 Decision Tree Learning Original slides by Raymond J. Mooney University of Texas at Austin.

Decision Trees, Part 1 Reading: Textbook, Chapter 6.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 6.2: Classification Rules Rodney Nielsen Many.

Decision Tree Learning Presented by Ping Zhang Nov. 26th, 2007.

Seminar on Machine Learning Rada Mihalcea Decision Trees Very short intro to Weka January 27, 2003.

Outline Decision tree representation ID3 learning algorithm Entropy, Information gain Issues in decision tree learning 2.

Friday’s Deliverable As a GROUP, you need to bring 2N+1 copies of your “initial submission” –This paper should be a complete version of your paper – something.

Decision Tree Pruning problem of overfit approaches

Data Mining CH6 Implementation: Real machine learning schemes(2) Reporter: H.C. Tsai.

Decision Tree Learning DA514 - Lecture Slides 2 Modified and expanded from: E. Alpaydin-ML (chapter 9) T. Mitchell-ML.

Review of Decision Tree Learning Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.

By Subhasis Dasgupta Asst Professor Praxis Business School, Kolkata Classification Modeling Decision Tree (Part 2)

Data Mining Practical Machine Learning Tools and Techniques Chapter 6.3: Association Rules Rodney Nielsen Many / most of these slides were adapted from:

CSE573 Autumn /11/98 Machine Learning Administrative –Finish this topic –The rest of the time is yours –Final exam Tuesday, Mar. 17, 2:30-4:20.

Decision Tree Learning

Decision trees (concept learnig)

Machine Learning Lecture 2: Decision Tree Learning.

Decision trees (concept learnig)

Decision Tree Learning

C4.5 algorithm Let the classes be denoted {C1, C2,…, Ck}. There are three possibilities for the content of the set of training samples T in the given node.

Ch9: Decision Trees 9.1 Introduction A decision tree:

Decision Tree Saed Sayad 9/21/2018.

Introduction to Data Mining, 2nd Edition by

Introduction to Data Mining, 2nd Edition by

Machine Learning Chapter 3. Decision Tree Learning

Decision Trees Decision tree representation ID3 learning algorithm

Machine Learning Chapter 3. Decision Tree Learning

Heaps By JJ Shepherd.

INTRODUCTION TO Machine Learning

Decision Trees Jeff Storey.

Presentation transcript:

Decision Tree Pruning Methods Validation set – withhold a subset (~1/3) of training data to use for pruning –Note: you should randomize the order of training examples

Reduced-Error Pruning Classify examples in validation set – some might be errors For each node: –Sum the errors over entire subtree –Calculate error on same example if converted to a leaf with majority class label Prune node with highest reduction in error Repeat until error no longer reduced

2- 2+,1- 3+, ,2- 2+,3- 4+,2- (code hint: design Node data structure to keep track of examples that pass through each node during classification)

Pessimistic Pruning Avoids needs to use validation set, can train on more examples Use conservative estimate of true error at each node, based on training examples “Continuity correction” to error rate at each node: add 1/2N to observed errors, for N the number of leaves in sub-tree Prune node unless est. errors of subtree is more than 1 standard error below est. for pruned: r’ subtree <r’ pruned -SE

Cost-Complexity Pruning On training examples, initial tree has no errors, but replacing subtrees with leaves increases errors “cost-complexity” – a measure of avg. error reduced per leaf Calculate number of errors for each node if collapsed to leaf compare to errors in leaves, taking into account more nodes used R(26,pruned)=15/200 R(26,subtree)=10/200 Cost-complexity is balanced when: R(n,pr)+  =R(n,su)+  N(su) 15/200+  =10/200+4   =0.0083

Calculate  for each node; prune node with smallest  Repeat, creating a series of trees T 0,T 1,T 2 … of decreasing size Pick tree with min error on validation set …or smallest tree within one standard error of minimum

Rule Post-Pruning Convert tree to rules (one for each path from root to a leaf) For each antecedent in a rule, remove it if error rate on validation set does not decrease Sort final rule set by accuracy Outlook=sunny ^ humidity=high -> No Outlook=sunny ^ humidity=normal -> Yes Outlook=overcast -> Yes Outlook=rain ^ wind=strong -> No Outlook=rain ^ wind=weak -> Yes Compare first rule to: Outlook=sunny->No Humidity=high->No Calculate accuracy of 3 rules based on validation set and pick best version.