1. 1 Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Modified by Longin Jan Latecki Some slides by Piyush Rai Intro AIDecision Trees

2. 2 Outline Decision Tree Representations –ID3 and C4.5 learning algorithms (Quinlan 1986) –CART learning algorithm (Breiman et al. 1985) Entropy, Information Gain Overfitting Intro AIDecision Trees

3. 3 Training Data Example: Goal is to Predict When This Player Will Play Tennis? Intro AIDecision Trees

4. 4Intro AIDecision Trees

5. 5Intro AIDecision Trees

6. 6Intro AIDecision Trees

7. 7Intro AIDecision Trees

8. 8 Learning Algorithm for Decision Trees What happens if features are not binary? What about regression? Intro AIDecision Trees

9. 9 Choosing the Best Attribute Intro AIDecision Trees - Many different frameworks for choosing BEST have been proposed! - We will look at Entropy Gain. Number + and – examples before and after a split. A1 and A2 are “attributes” (i.e. features or inputs).

10. 10 Entropy Intro AIDecision Trees

11. 11Intro AIDecision Trees Entropy is like a measure of impurity…

12. 12 Entropy Intro AIDecision Trees

13. Intro AIDecision Trees13

14. 14 Information Gain Intro AIDecision Trees

15. Intro AIDecision Trees15

16. Intro AIDecision Trees16

17. Intro AIDecision Trees17

18. 18 Training Example Intro AIDecision Trees

19. 19 Selecting the Next Attribute Intro AIDecision Trees

20. 20Intro AIDecision Trees

21. 21 Non-Boolean Features Features with multiple discrete values –Multi-way splits –Test for one value versus the rest –Group values into disjoint sets Real-valued features –Use thresholds Regression –Splits based on mean squared error metric Intro AIDecision Trees

22. 22 Hypothesis Space Search Intro AIDecision Trees You do not get the globally optimal tree! - Search space is exponential.

23. 23 Overfitting Intro AIDecision Trees

24. 24 Overfitting in Decision Trees Intro AIDecision Trees

25. 25 Validation Data is Used to Control Overfitting Prune tree to reduce error on validation set Intro AIDecision Trees

26. Homework Which feature will be at the root node of the decision tree trained for the following data? In other words which attribute makes a person most attractive? Intro AIDecision Trees26 HeightHairEyesAttractive? smallblondebrownNo talldarkbrownNo tallblondeblueYes talldarkBlueNo smalldarkBlueNo tallredBlueYes tallblondebrownNo smallblondeblueYes