1. © Vipin Kumar CSci 8980 Fall 2002 1 CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance Computing Research Center Department of Computer Science University of Minnesota http://www.cs.umn.edu/~kumar

2. © Vipin Kumar CSci 8980 Fall 2002 2 Splitting Criterion l Gini Index l Entropy and Information Gain l Misclassification error

3. © Vipin Kumar CSci 8980 Fall 2002 3 Splitting Criterion: GINI l Gini Index for a given node t : (NOTE: p( j | t) is the relative frequency of class j at node t). –Measures impurity of a node.  Maximum (1 - 1/n c ) when records are equally distributed among all classes, implying least interesting information  Minimum (0.0) when all records belong to one class, implying most interesting information

4. © Vipin Kumar CSci 8980 Fall 2002 4 Examples for computing GINI P(C1) = 0/6 = 0 P(C2) = 6/6 = 1 Gini = 1 – P(C1) 2 – P(C2) 2 = 1 – 0 – 1 = 0 P(C1) = 1/6 P(C2) = 5/6 Gini = 1 – (1/6) 2 – (5/6) 2 = 0.278 P(C1) = 2/6 P(C2) = 4/6 Gini = 1 – (2/6) 2 – (4/6) 2 = 0.444

5. © Vipin Kumar CSci 8980 Fall 2002 5 Splitting Based on GINI l Used in CART, SLIQ, SPRINT. l Splitting Criterion: Minimize Gini Index of the Split. l When a node p is split into k partitions (children), the quality of split is computed as, where,n i = number of records at child i, n = number of records at node p.

6. © Vipin Kumar CSci 8980 Fall 2002 6 Binary Attributes: Computing GINI Index l Splits into two partitions l Effect of Weighing partitions: –Larger and Purer Partitions are sought for. B? YesNo Node N1Node N2 Gini(N1) = 1 – (5/6) 2 – (1/6) 2 = 0.278 Gini(N2) = 1 – (2/6) 2 – (4/6) 2 = 0.444 Gini(Children) = 6/12 * 0.278 + 6/12 * 0.444 = 0.361

7. © Vipin Kumar CSci 8980 Fall 2002 7 Categorical Attributes: Computing Gini Index l For each distinct value, gather counts for each class in the dataset l Use the count matrix to make decisions Multi-way splitTwo-way split (find best partition of values)

8. © Vipin Kumar CSci 8980 Fall 2002 8 Continuous Attributes: Computing Gini Index l Use Binary Decisions based on one value l Several Choices for the splitting value –Number of possible splitting values = Number of distinct values l Each splitting value has a count matrix associated with it –Class counts in each of the partitions, A < v and A  v l Simple method to choose best v –For each v, scan the database to gather count matrix and compute its Gini index –Computationally Inefficient! Repetition of work.

9. © Vipin Kumar CSci 8980 Fall 2002 9 Continuous Attributes: Computing Gini Index... l For efficient computation: for each attribute, –Sort the attribute on values –Linearly scan these values, each time updating the count matrix and computing gini index –Choose the split position that has the least gini index Split Positions Sorted Values

10. © Vipin Kumar CSci 8980 Fall 2002 10 Alternative Splitting Criteria based on INFO l Entropy at a given node t: (NOTE: p( j | t) is the relative frequency of class j at node t). –Measures homogeneity of a node.  Maximum (log n c ) when records are equally distributed among all classes implying least information  Minimum (0.0) when all records belong to one class, implying most information –Entropy based computations are similar to the GINI index computations

11. © Vipin Kumar CSci 8980 Fall 2002 11 Examples for computing Entropy P(C1) = 0/6 = 0 P(C2) = 6/6 = 1 Entropy = – 0 log 0 – 1 log 1 = – 0 – 0 = 0 P(C1) = 1/6 P(C2) = 5/6 Entropy = – (1/6) log 2 (1/6) – (5/6) log 2 (1/6) = 0.65 P(C1) = 2/6 P(C2) = 4/6 Entropy = – (2/6) log 2 (2/6) – (4/6) log 2 (4/6) = 0.92

12. © Vipin Kumar CSci 8980 Fall 2002 12 Splitting Based on INFO... l Information Gain: Parent Node, p is split into k partitions; n i is number of records in partition i –Measures Reduction in Entropy achieved because of the split. Choose the split that achieves most reduction (maximizes GAIN) –Used in ID3 and C4.5

13. © Vipin Kumar CSci 8980 Fall 2002 13 Splitting Based on INFO... l Disadvantage of Information Gain: –Tends to prefer splits that result in large number of partitions, each being small but pure. l Gain Ratio: Parent Node, p is split into k partitions n i is the number of records in partition i –Adjusts Information Gain by the entropy of the partitioning (SplitINFO). Higher entropy partitioning (large number of small partitions) is penalized! –Used in C4.5

14. © Vipin Kumar CSci 8980 Fall 2002 14 Splitting Criteria based on Classification Error l Classification error at a node t : l Measures misclassification error made by a node.  Maximum (1 - 1/n c ) when records are equally distributed among all classes, implying least interesting information  Minimum (0.0) when all records belong to one class, implying most interesting information

15. © Vipin Kumar CSci 8980 Fall 2002 15 Examples for Computing Error P(C1) = 0/6 = 0 P(C2) = 6/6 = 1 Error = 1 – max (0, 1) = 1 – 1 = 0 P(C1) = 1/6 P(C2) = 5/6 Error = 1 – max (1/6, 5/6) = 1 – 5/6 = 1/6 P(C1) = 2/6 P(C2) = 4/6 Error = 1 – max (2/6, 4/6) = 1 – 4/6 = 1/3

16. © Vipin Kumar CSci 8980 Fall 2002 16 Comparison among Splitting Criteria For a 2-class problem:

17. © Vipin Kumar CSci 8980 Fall 2002 17 Misclassification Error vs Gini A? YesNo Node N1Node N2 Gini(N1) = 1 – (3/3) 2 – (0/3) 2 = 0 Gini(N2) = 1 – (4/7) 2 – (3/7) 2 = 0.489 Gini(Children) = 3/10 * 0 + 7/10 * 0.489 = 0.342 Gini improves !!

18. © Vipin Kumar CSci 8980 Fall 2002 18 C4.5 l Simple depth-first construction. l Uses Information Gain l Sorts Continuous Attributes at each node. l Needs entire data to fit in memory. l Unsuitable for Large Datasets. –Needs out-of-core sorting.

19. © Vipin Kumar CSci 8980 Fall 2002 19 Practical Challenges of Classification l Overfitting –Model performs well on training, but poorly on test data. l Missing Values l Data Heterogeneity l Costs –Costs for measuring attributes –Costs of misclassification

20. © Vipin Kumar CSci 8980 Fall 2002 20 Overfitting (Example) 500 circular and 500 triangular data points. Circular points: 0.5  sqrt(x 1 2 +x 2 2 )  1 Triangular points: sqrt(x 1 2 +x 2 2 ) > 0.5 or sqrt(x 1 2 +x 2 2 ) < 1

21. © Vipin Kumar CSci 8980 Fall 2002 21 Overfitting… Overfitting

22. © Vipin Kumar CSci 8980 Fall 2002 22 Overfitting due to Noise Decision boundary is distorted by noise point

23. © Vipin Kumar CSci 8980 Fall 2002 23 Overfitting due to Insufficient Examples Insufficient number of points may cause the decision boundary to change