© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Classification: Definition l Given a collection of records (training set) l Find a model for class attribute as a function of the values of other attributes. l Goal: previously unseen records should be assigned a class as accurately as possible. –A test set is used to determine the accuracy.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Illustrating Classification Task

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Examples of Classification Task l Predicting tumor cells as benign or malignant l Classifying credit card transactions as legitimate or fraudulent l Classifying secondary structures of protein as alpha-helix, beta-sheet, or random coil l Categorizing news stories as finance, weather, entertainment, sports, etc

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Classification Techniques l Decision Tree based Methods l Rule-based Methods l Memory based reasoning l Neural Networks l Naïve Bayes and Bayesian Belief Networks l Support Vector Machines

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Example of a Decision Tree categorical continuous class Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Splitting Attributes Training Data Model: Decision Tree

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Another Example of Decision Tree categorical continuous class MarSt Refund TaxInc YES NO Yes No Married Single, Divorced < 80K> 80K There could be more than one tree that fits the same data!

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Decision Tree Classification Task Decision Tree

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Apply Model to Test Data Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Test Data Start from the root of tree.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Apply Model to Test Data Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Test Data

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Apply Model to Test Data Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Test Data

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Apply Model to Test Data Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Test Data

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Apply Model to Test Data Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Test Data

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Apply Model to Test Data Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Test Data Assign Cheat to “No”

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Decision Tree Classification Task Decision Tree

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Decision Tree Induction l Many Algorithms: –Hunt’s Algorithm (one of the earliest) –CART –ID3, C4.5 –SLIQ,SPRINT

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ General Structure of Hunt’s Algorithm l D t = set of training records of node t l General Procedure: –If D t only records of same class y t  t leaf node labeled as y t –Else: use an attribute test to split the data. l Recursively apply the procedure to each subset. DtDt ?

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Hunt’s Algorithm Don’t Cheat Refund Don’t Cheat Don’t Cheat YesNo Refund Don’t Cheat YesNo Marital Status Don’t Cheat Single, Divorced Married Taxable Income Don’t Cheat < 80K>= 80K Refund Don’t Cheat YesNo Marital Status Don’t Cheat Single, Divorced Married

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Tree Induction l Greedy strategy. –Split the records based on an attribute test that optimizes a local criterion. l Issues –Determine how to split the records  How to specify the attribute test condition?  How to determine the best split? –Determine when to stop splitting

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Tree Induction l Greedy strategy. –Split the records based on an attribute test that optimizes certain criterion. l Issues –Determine how to split the records  How to specify the attribute test condition?  How to determine the best split? –Determine when to stop splitting

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ How to Specify Test Condition? l Depends on attribute types –Nominal(No order; e.g., Country) –Ordinal(Discrete, order; e.g., S,M,L,XL) –Continuous (Ordered, cont.; e.g., temperature) l Depends on number of ways to split –2-way split –Multi-way split

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Splitting Based on Nominal Attributes l Multi-way split: Use as many partitions as distinct values. l Binary split: Divides values into two subsets. Need to find optimal partitioning. CarType Family Sports Luxury CarType {Family, Luxury} {Sports} CarType {Sports, Luxury} {Family} OR

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Splitting Based on Continuous Attributes l Different ways of handling –Discretization to form an ordinal categorical attribute  Static – discretize once at the beginning  Dynamic – ranges can be found by equal interval bucketing, equal frequency bucketing (percentiles), or clustering. –Binary Decision: (A < v) or (A  v)  consider all possible splits and finds the best cut  can be more compute intensive

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Splitting Based on Continuous Attributes

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Tree Induction l Greedy strategy. –Split the records based on an attribute test that optimizes certain criterion. l Issues –Determine how to split the records  How to specify the attribute test condition?  How to determine the best split? –Determine when to stop splitting

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ How to determine the Best Split Before Splitting: 10 records of class 0, 10 records of class 1 Which test condition is the best?

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ How to determine the Best Split l Greedy approach: –Nodes with homogeneous class distribution are preferred l Need a measure of node impurity: Non-homogeneous, High degree of impurity Homogeneous, Low degree of impurity

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Measures of Node Impurity l Gini Index l Entropy l Misclassification error

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ How to Find the Best Split B? YesNo Node N3Node N4 A? YesNo Node N1Node N2 Before Splitting: M0 M1 M2M3M4 M12 M34 Gain = M0 – M12 vs M0 – M34

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Measure of Impurity: GINI l Gini Index for a given node t : (NOTE: p( j | t) is the relative frequency of class j at node t). –Maximum (1 - 1/n c ) when records are equally distributed among all classes, implying least interesting information –Minimum (0.0) when all records belong to one class, implying most interesting information

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Examples for computing GINI P(C1) = 0/6 = 0 P(C2) = 6/6 = 1 Gini = 1 – P(C1) 2 – P(C2) 2 = 1 – 0 – 1 = 0 P(C1) = 1/6 P(C2) = 5/6 Gini = 1 – (1/6) 2 – (5/6) 2 = P(C1) = 2/6 P(C2) = 4/6 Gini = 1 – (2/6) 2 – (4/6) 2 = 0.444

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Splitting Based on GINI l Used in CART, SLIQ, SPRINT. l When a node p is split into k partitions (children), the quality of split is computed as, where,n i = number of records at child i, n = number of records at node p.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Binary Attributes: Computing GINI Index l Splits into two partitions l Effect of Weighing partitions: –Larger and Purer Partitions are sought for. B? YesNo Node N1Node N2 Gini(N1) = 1 – (5/6) 2 – (2/6) 2 = Gini(N2) = 1 – (1/6) 2 – (4/6) 2 = Gini(Children) = 7/12 * /12 * = 0.333

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Tree Induction l Greedy strategy. –Split the records based on an attribute test that optimizes certain criterion. l Issues –Determine how to split the records  How to specify the attribute test condition?  How to determine the best split? –Determine when to stop splitting

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Stopping Criteria for Tree Induction l Stop expanding a node when all the records belong to the same class l Stop expanding a node when all the records have similar attribute values l Early termination (to be discussed later)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Decision Tree Based Classification l Advantages: –Inexpensive to construct –Extremely fast at classifying unknown records –Easy to interpret for small-sized trees –Accuracy is comparable to other classification techniques for many simple data sets

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Practical Issues of Classification l Underfitting and Overfitting l Missing Values l Costs of Classification

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Underfitting and Overfitting Overfitting Underfitting: when model is too simple, both training and test errors are large Underfitting

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Overfitting due to Noise Decision boundary is distorted by noise point

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Overfitting due to Insufficient Examples Lack of data points in the lower half of the diagram makes it difficult to predict correctly the class labels of that region - Insufficient number of training records in the region causes the decision tree to predict the test examples using other training records that are irrelevant to the classification task

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Notes on Overfitting l Overfitting results in decision trees that are more complex than necessary l Training error no longer provides a good estimate of how well the tree will perform on previously unseen records l Need new ways for estimating errors

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ How to Address Overfitting l Pre-Pruning (Early Stopping Rule) –Stop the algorithm before it becomes a fully- grown tree  Stop if number of instances is less than some user- specified threshold  Stop if class distribution of instances are independent of the available features (e.g., using  2 test)  Stop if expanding the current node does not improve impurity measures (e.g., Gini or information gain).

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ How to Address Overfitting… l Post-pruning –Grow decision tree to its entirety –Trim the nodes of the decision tree in a bottom-up fashion –If generalization error improves after trimming, replace sub-tree by a leaf node. –Class label of leaf node is determined from majority class of instances in the sub-tree

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Model Evaluation l Metrics for Performance Evaluation –How to evaluate the performance of a model? l Methods for Performance Evaluation –How to obtain reliable estimates? l Methods for Model Comparison –How to compare the relative performance among competing models?

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Metrics for Performance Evaluation l Focus on the predictive capability of a model –Rather than how fast it takes to classify or build models, scalability, etc. l Confusion Matrix: PREDICTED CLASS ACTUAL CLASS Class=YesClass=No Class=Yesab Class=Nocd a: TP (true positive) b: FN (false negative) c: FP (false positive) d: TN (true negative)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Metrics for Performance Evaluation… l Most widely-used metric: PREDICTED CLASS ACTUAL CLASS Class=YesClass=No Class=Yesa (TP) b (FN) Class=Noc (FP) d (TN)

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Limitation of Accuracy l Consider a 2-class problem –Number of Class 0 examples = 9990 –Number of Class 1 examples = 10 l If model predicts everything to be class 0, accuracy is 9990/10000 = 99.9 % –Accuracy is misleading because model does not detect any class 1 example

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Cost-Sensitive Measures l Precision is biased towards C(Yes|Yes) & C(Yes|No) l Recall is biased towards C(Yes|Yes) & C(No|Yes) l F-measure is biased towards all except C(No|No)