1. Classification Continued
Decision Trees
Classification
Continued
Dr Eamonn Keogh
Computer Science & Engineering Department University of California - Riverside Riverside,CA

2. Lets review the classification techniques we have seen so far, in terms of decision surfaces.

3. -3 -2 -1 1 2 3 4 5 6 7

4. -3 -2 -1 1 2 3 4 5 6 7
Linear Classifier

5. Nearest Neighbor Classifier

6. 4 3 2 1 -1 -2 -2 -1 1 2 3 4 5 6

7. Decision Tree Classification
A flow-chart-like tree structure
Internal node denotes a test on an attribute
Branch represents an outcome of the test
Leaf nodes represent class labels or class distribution
Decision tree generation consists of two phases
Tree construction
At start, all the training examples are at the root
Partition examples recursively based on selected attributes
Tree pruning
Identify and remove branches that reflect noise or outliers
Use of decision tree: Classifying an unknown sample
Test the attribute values of the sample against the decision tree

8. Decision Tree Example I
We have above data in our database, based upon this data, we want to predict if potential customers are likely to buy a computer.
For example: will Joe, a 25 year old lumberjack with medium income and a fair credit rating buy a PC?

9. Decision Tree Example II
Joe, a 25 year old lumberjack with medium income and a fair credit rating.
?, <=30, medium, no, fair
Age?
<=30
Student?
31to40
>40
Yes
CreditRating? no
yes
excellent
fair

10. How do we construct the decision tree?
Basic algorithm (a greedy algorithm)
Tree is constructed in a top-down recursive divide-and-conquer manner
At start, all the training examples are at the root
Attributes are categorical (if continuous-valued, they can be discretized in advance)
Examples are partitioned recursively based on selected attributes.
Test attributes are selected on the basis of a heuristic or statistical measure (e.g., information gain)
Conditions for stopping partitioning
All samples for a given node belong to the same class
There are no remaining attributes for further partitioning – majority voting is employed for classifying the leaf
There are no samples left

11. if blood sugar > 3.5 then class = sick else class = healthy10 1 2 3 4 5 6 7 8 9
Imagine this dataset shows two classes of people, healthy and sick.
The X-axis shows their blood sugar count, the Y axis shows their white cell count.
We want to find the single best rule of the form
if somefeature > somevalue then
class = sick
else
class = healthy
if blood sugar > 3.5 then class = sick else class = healthy

12. 10 1 2 3 4 5 6 7 8 9
Blood Sugar > 3.5? no
yes
Healthy
sick

13. 10 1 2 3 4 5 6 7 8 9

14. We have only informally shown how the decision tree chooses the splitting point for continuous attributes.
How do we choose a splitting criteria for nominal or Boolean attributes?
We want to find the single best rule of the form
if somefeature = somevalue then
class = sick
else
class = healthy M
Gender F
Height

16. Example of a problem that decision trees do poorly on.

17. We have now seen several classification algorithms
We have now seen several classification algorithms. How should we compare them?
Predictive accuracy
Speed and scalability
time to construct the model
time to use the model
Robustness
handling noise and missing values
Scalability
efficiency in disk-resident databases
Interpretability:
understanding and insight provided by the model.

18. What happens if we run out of features to test before correctly partitioning the test set?
Weight
<=30
Blood PH
< 7? 31 to 40
>40
Yes
Height > 178? no
yes
< 5?
Here we have a dataset with 3 features.
Weight
Blood PH
Height
We are trying to classify people into two classes, yes or no (ie yes, they will get sick or no they won’t).
Most items are classified, but 28 individuals remain unclassified after using all features...
18 no
10 yes
18 healthy
12 sick

19. Feature Generation 10 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 Weight
Height
BMI
BMI=kg/m2

20. Feature Generation Case Study
Suppose we have the following two classes..
Class A: 100 random coin tosses
Class B: A human “faking” 100 random coin tosses
A ….
A ….
B ….
B ….
A ….

21. What splitting criteria should we use?

22. How many bits do I need to label all the objects in this box?
How many bits do I need to label all the objects in these boxes?

23. Information Gain as A Splitting Criteria
Select the attribute with the highest information gain (information gain is the expected reduction in entropy).
Assume there are two classes, P and N
Let the set of examples S contain p elements of class P and n elements of class N
The amount of information, needed to decide if an arbitrary example in S belongs to P or N is defined as
0 log(0) is defined as 0

24. Information Gain in Decision Tree Induction
Assume that using attribute A, a current set will be partitioned into some number of child sets
The encoding information that would be gained by branching on A
Note: entropy is at its minimum if the collection of objects is completely uniform

25. log(1) = 0 Entropy(9 ,5 ) = -(9/14)log2(9/14) - (5/14)log2(5/14)=
Entropy(9 ,0 ) = -(9/9)log2(9/9) - (0/9)log2(0/9)
= 0
Entropy(0 ,5 ) = -(0/5)log2(0/5) - (5/5)log2(5/5)
= 0
log(1) = 0

26. ) - Gain(A) = + = 0.940 log(1) = 0 Entropy(9 ,5 ) ( Entropy(9 ,0 )

27. Avoiding Overfitting in Classification
The generated tree may overfit the training data
Too many branches, some may reflect anomalies due to noise or outliers
Result is in poor accuracy for unseen samples
Two approaches to avoid overfitting
Prepruning: Halt tree construction early—do not split a node if this would result in the goodness measure falling below a threshold
Difficult to choose an appropriate threshold
Postpruning: Remove branches from a “fully grown” tree—get a sequence of progressively pruned trees
Use a set of data different from the training data to decide which is the “best pruned tree”

28. Approaches to Determine the Final Tree Size
Separate training (2/3) and testing (1/3) sets
Use cross validation, e.g., 10-fold cross validation
Use all the data for training
but apply a statistical test (e.g., chi-square) to estimate whether expanding or pruning a node may improve the entire distribution

29. Feature Selection
One of the nice features of decision trees is that they automatically discover the best features to use (the ones near the top of the tree), and which features are irrelevant for the problem (the features which are no used).
How do we decide which features to use for nearest neighbor, or the linear classifier?
Suppose we are trying to decide if tomorrow is a good day to play tennis, based on the temperature, the windspeed, the humidity and the outlook…
We could use just the temperature, or just {temperature, windspeed} or just {…} This sounds like a search problem!

30. Forward Selection
Backward Elimination
Bi-directional Search

31. -3 -2 -1 1 2 3 4 5 6 7
Nearest
Neighbor Classifier