Presentation is loading. Please wait.

Presentation is loading. Please wait.

DECISION TREES David Kauchak CS 451 – Fall 2013. Admin Assignment 1 available and is due on Friday (printed out at the beginning of class) Door code for.

Published byAustin Fisher Modified over 9 years ago

Similar presentations

Presentation on theme: "DECISION TREES David Kauchak CS 451 – Fall 2013. Admin Assignment 1 available and is due on Friday (printed out at the beginning of class) Door code for."— Presentation transcript:

1 DECISION TREES David Kauchak CS 451 – Fall 2013

2 Admin Assignment 1 available and is due on Friday (printed out at the beginning of class) Door code for MBH632 Keep up with the reading Videos

3 Quick refresher from last time…

4 Representing examples examples What is an example? How is it represented?

5 Features examples f 1, f 2, f 3, …, f n features f 1, f 2, f 3, …, f n How our algorithms actually “view” the data Features are the questions we can ask about the examples

6 Features examples red, round, leaf, 3oz, … features How our algorithms actually “view” the data Features are the questions we can ask about the examples green, round, no leaf, 4oz, … yellow, curved, no leaf, 4oz, … green, curved, no leaf, 5oz, …

7 Classification revisited red, round, leaf, 3oz, … green, round, no leaf, 4oz, … yellow, curved, no leaf, 4oz, … green, curved, no leaf, 5oz, … label apple banana examples model/ classifier learn During learning/training/induction, learn a model of what distinguishes apples and bananas based on the features

8 Classification revisited red, round, no leaf, 4oz, … model/ classifier The model can then classify a new example based on the features predict Apple or banana?

9 Classification revisited red, round, no leaf, 4oz, … model/ classifier The model can then classify a new example based on the features predict Apple Why?

10 Classification revisited red, round, leaf, 3oz, … green, round, no leaf, 4oz, … yellow, curved, no leaf, 4oz, … green, curved, no leaf, 5oz, … label apple banana examples Training data red, round, no leaf, 4oz, … ? Test set

11 Classification revisited red, round, leaf, 3oz, … green, round, no leaf, 4oz, … yellow, curved, no leaf, 4oz, … green, curved, no leaf, 5oz, … label apple banana examples Training data red, round, no leaf, 4oz, … ? Learning is about generalizing from the training data Test set What does this assume about the training and test set?

12 Past predicts future Training dataTest set

13 Past predicts future Training dataTest set Not always the case, but we’ll often assume it is!

14 Past predicts future Training dataTest set Not always the case, but we’ll often assume it is!

15 More technically… We are going to use the probabilistic model of learning There is some probability distribution over example/label pairs called the data generating distribution Both the training data and the test set are generated based on this distribution What is a probability distribution?

16 Probability distribution Describes how likely (i.e. probable) certain events are

17 Probability distribution Training data round apples curved bananas apples with leaves … High probabilityLow probability curved apples red bananas yellow apples …

18 data generating distribution Training dataTest set data generating distribution

19 Training dataTest set data generating distribution

20 Training dataTest set data generating distribution

21 A sample data set FeaturesLabel HourWeatherAccidentStallCommute 8 AMSunnyNo Long 8 AMCloudyNoYesLong 10 AMSunnyNo Short 9 AMRainyYesNoLong 9 AMSunnyYes Long 10 AMSunnyNo Short 10 AMCloudyNo Short 9 AMSunnyYesNoLong 10 AMCloudyYes Long 10 AMRainyNo Short 8 AMCloudyYesNoLong 9 AMRainyNo Short 8 AM, Rainy, Yes, No? 10 AM, Rainy, No, No? Can you describe a “model” that could be used to make decisions in general?

22 Short Decision trees Leave At Stall?Accident? 10 AM 9 AM 8 AM Long ShortLong NoYes NoYes Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes

23 Short Decision trees Leave At Stall?Accident? 10 AM 9 AM 8 AM Long ShortLong NoYes NoYes Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes Leave = 8 AM Weather = Rainy Accident = Yes Stall = No

24 Short Decision trees Leave At Stall?Accident? 10 AM 9 AM 8 AM Long ShortLong NoYes NoYes Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes Leave = 8 AM Weather = Rainy Accident = Yes Stall = No

25 Short Decision trees Leave At Stall?Accident? 10 AM 9 AM 8 AM Long ShortLong NoYes NoYes Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes Leave = 10 AM Weather = Rainy Accident = No Stall = No

26 Short Decision trees Leave At Stall?Accident? 10 AM 9 AM 8 AM Long ShortLong NoYes NoYes Tree with internal nodes labeled by features Branches are labeled by tests on that feature Leaves labeled with classes Leave = 10 AM Weather = Rainy Accident = No Stall = No

27 To ride or not to ride, that is the question… Build a decision tree TerrainUnicycle- type WeatherGo-For-Ride? TrailNormalRainyNO RoadNormalSunnyYES TrailMountainSunnyYES RoadMountainRainyYES TrailNormalSnowyNO RoadNormalRainyYES RoadMountainSnowyYES TrailNormalSunnyNO RoadNormalSnowyNO TrailMountainSnowyYES

28 Recursive approach Base case: If all data belong to the same class, create a leaf node with that label Otherwise: - calculate the “score” for each feature if we used it to split the data - pick the feature with the highest score, partition the data based on that data value and call recursively

29 Snowy Partitioning the data TerrainUnicycle- type WeatherGo-For- Ride? TrailNormalRainyNO RoadNormalSunnyYES TrailMountainSunnyYES RoadMountainRainyYES TrailNormalSnowyNO RoadNormalRainyYES RoadMountainSnowyYES TrailNormalSunnyNO RoadNormalSnowyNO TrailMountainSnowyYES Terrain Road Trail YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 Weather Rainy Sunny YES: 2 NO: 1 YES: 2 NO: 1 YES: 2 NO: 2

30 Partitioning the data Terrain Road Trail YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 Snowy Weather Rainy Sunny YES: 2 NO: 1 YES: 2 NO: 1 YES: 2 NO: 2 calculate the “score” for each feature if we used it to split the data What score should we use? If we just stopped here, which tree would be best? How could we make these into decision trees?

31 Decision trees Terrain Road Trail YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 Snowy Weather Rainy Sunny YES: 2 NO: 1 YES: 2 NO: 1 YES: 2 NO: 2 How could we make these into decision trees?

32 Decision trees Terrain Road Trail YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 Snowy Weather Rainy Sunny YES: 2 NO: 1 YES: 2 NO: 1 YES: 2 NO: 2

33 Decision trees Terrain Road Trail YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 Snowy Weather Rainy Sunny YES: 2 NO: 1 YES: 2 NO: 1 YES: 2 NO: 2 Training error: the average error over the training set For classification, the most common “error” is the number of mistakes Training error for each of these?

34 Decision trees Terrain Road Trail YES: 4 NO: 1 YES: 2 NO: 3 Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 Snowy Weather Rainy Sunny YES: 2 NO: 1 YES: 2 NO: 1 YES: 2 NO: 2 Training error: the average error over the training set 3/102/104/10

35 Recurse Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 TerrainUnicycle- type WeatherGo-For- Ride? TrailNormalRainyNO RoadNormalSunnyYES TrailNormalSnowyNO RoadNormalRainyYES TrailNormalSunnyNO RoadNormalSnowyNO TerrainUnicycle- type WeatherGo-For- Ride? TrailMountainSunnyYES RoadMountainRainyYES RoadMountainSnowyYES TrailMountainSnowyYES

36 Recurse Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 TerrainUnicycle- type WeatherGo-For- Ride? TrailNormalRainyNO RoadNormalSunnyYES TrailNormalSnowyNO RoadNormalRainyYES TrailNormalSunnyNO RoadNormalSnowyNO Snowy Terrain Road Trail YES: 2 NO: 1 YES: 0 NO: 3 Weather Rainy Sunny YES: 1 NO: 1 YES: 1 NO: 1 YES: 0 NO: 2

37 Recurse Unicycle Mountain Normal YES: 4 NO: 0 YES: 2 NO: 4 TerrainUnicycle- type WeatherGo-For- Ride? TrailNormalRainyNO RoadNormalSunnyYES TrailNormalSnowyNO RoadNormalRainyYES TrailNormalSunnyNO RoadNormalSnowyNO Snowy Terrain Road Trail YES: 2 NO: 1 YES: 0 NO: 3 Weather Rainy Sunny YES: 1 NO: 1 YES: 1 NO: 1 YES: 0 NO: 2 1/6 2/6

38 Recurse Unicycle Mountain Normal YES: 4 NO: 0 TerrainUnicycle- type WeatherGo-For- Ride? RoadNormalSunnyYES RoadNormalRainyYES RoadNormalSnowyNO Terrain Road Trail YES: 2 NO: 1 YES: 0 NO: 3

39 Recurse Unicycle Mountain Normal YES: 4 NO: 0 Terrain Road Trail YES: 0 NO: 3 Snowy Weather Rainy Sunny YES: 1 NO: 1 YES: 1 NO: 0 YES: 0 NO: 1

Download ppt "DECISION TREES David Kauchak CS 451 – Fall 2013. Admin Assignment 1 available and is due on Friday (printed out at the beginning of class) Door code for."

Similar presentations

Knowledge Acquisition and Modelling

Knowledge Acquisition and Modelling
Data Mining Lecture 9.

Data Mining Lecture 9.
Decision Trees Decision tree representation ID3 learning algorithm

Decision Trees Decision tree representation ID3 learning algorithm
Imbalanced data David Kauchak CS 451 – Fall 2013.

Imbalanced data David Kauchak CS 451 – Fall 2013.
Huffman code and ID3 Prof. Sin-Min Lee Department of Computer Science.

Huffman code and ID3 Prof. Sin-Min Lee Department of Computer Science.
Decision Tree Approach in Data Mining

Decision Tree Approach in Data Mining
Non-Metric Methods: Decision Trees Alexandros Potamianos Dept of ECE, Tech. Univ. of Crete Fall 2004-2005.

Non-Metric Methods: Decision Trees Alexandros Potamianos Dept of ECE, Tech. Univ. of Crete Fall
PERCEPTRON LEARNING David Kauchak CS 451 – Fall 2013.

PERCEPTRON LEARNING David Kauchak CS 451 – Fall 2013.
Classification Techniques: Decision Tree Learning

Classification Techniques: Decision Tree Learning
PROBABILISTIC MODELS David Kauchak CS451 – Fall 2013.

PROBABILISTIC MODELS David Kauchak CS451 – Fall 2013.
PROBABILISTIC MODELS David Kauchak CS451 – Fall 2013.

PROBABILISTIC MODELS David Kauchak CS451 – Fall 2013.
Decision Trees Jeff Storey. Overview What is a Decision Tree Sample Decision Trees How to Construct a Decision Tree Problems with Decision Trees Decision.

Decision Trees Jeff Storey. Overview What is a Decision Tree Sample Decision Trees How to Construct a Decision Tree Problems with Decision Trees Decision.
1 18.3 Decision Trees Instructor: Qiang Yang Hong Kong University of Science and Technology Thanks: Eibe Frank and Jiawei Han.

Decision Trees Instructor: Qiang Yang Hong Kong University of Science and Technology Thanks: Eibe Frank and Jiawei Han.
Assuming normally distributed data! Naïve Bayes Classifier.

Assuming normally distributed data! Naïve Bayes Classifier.
Ensemble Learning: An Introduction

Ensemble Learning: An Introduction
Induction of Decision Trees

Induction of Decision Trees
Decision Trees an Introduction.

Decision Trees an Introduction.
Classification.

Classification.
Machine Learning Lecture 10 Decision Trees G53MLE Machine Learning Dr Guoping Qiu1.

Machine Learning Lecture 10 Decision Trees G53MLE Machine Learning Dr Guoping Qiu1.
Ensemble Learning (2), Tree and Forest

Ensemble Learning (2), Tree and Forest

Similar presentations

Ads by Google