# Iterative Dichotomiser 3 (ID3) Algorithm

## Presentation on theme: "Iterative Dichotomiser 3 (ID3) Algorithm"— Presentation transcript:

Iterative Dichotomiser 3 (ID3) Algorithm
Medha Pradhan CS 157B, Spring 2007

Agenda Basics of Decision Tree Introduction to ID3
Entropy and Information Gain Two Examples

Basics What is a decision tree?
A tree where each branching (decision) node represents a choice between 2 or more alternatives, with every branching node being part of a path to a leaf node Decision node: Specifies a test of some attribute Leaf node: Indicates classification of an example

ID3 Invented by J. Ross Quinlan
Employs a top-down greedy search through the space of possible decision trees. Greedy because there is no backtracking. It picks highest values first. Select attribute that is most useful for classifying examples (attribute that has the highest Information Gain).

Entropy Entropy measures the impurity of an arbitrary collection of examples. For a collection S, entropy is given as: For a collection S having positive and negative examples Entropy(S) = -p+log2p+ - p-log2p- where p+ is the proportion of positive examples and p- is the proportion of negative examples In general, Entropy(S) = 0 if all members of S belong to the same class. Entropy(S) = 1 (maximum) when all members are split equally.

Information Gain Measures the expected reduction in entropy. The higher the IG, more is the expected reduction in entropy. where Values(A) is the set of all possible values for attribute A, Sv is the subset of S for which attribute A has value v.

Example 1 Sample training data to determine whether an animal lays eggs. Independent/Condition attributes Dependent/Decision attributes Animal Warm-blooded Feathers Fur Swims Lays Eggs Ostrich Yes No Crocodile Raven Albatross Dolphin Koala

Entropy(4Y,2N): -(4/6)log2(4/6) – (2/6)log2(2/6)
= Now, we have to find the IG for all four attributes Warm-blooded, Feathers, Fur, Swims

For attribute ‘Warm-blooded’:
Values(Warm-blooded) : [Yes,No] S = [4Y,2N] SYes = [3Y,2N] E(SYes) = SNo = [1Y,0N] E(SNo) = 0 (all members belong to same class) Gain(S,Warm-blooded) = – [(5/6)* (1/6)*0] = For attribute ‘Feathers’: Values(Feathers) : [Yes,No] SYes = [3Y,0N] E(SYes) = 0 SNo = [1Y,2N] E(SNo) = Gain(S,Feathers) = – [(3/6)*0 + (3/6)* ] =

For attribute ‘Fur’: Values(Fur) : [Yes,No] S = [4Y,2N] SYes = [0Y,1N] E(SYes) = 0 SNo = [4Y,1N] E(SNo) = Gain(S,Fur) = – [(1/6)*0 + (5/6)*0.7219] = For attribute ‘Swims’: Values(Swims) : [Yes,No] SYes = [1Y,1N] E(SYes) = 1 (equal members in both classes) SNo = [3Y,1N] E(SNo) = Gain(S,Swims) = – [(2/6)*1 + (4/6)* ] =

Gain(S,Warm-blooded) = 0.10916 Gain(S,Feathers) = 0.45914
Gain(S,Fur) = Gain(S,Swims) = Gain(S,Feathers) is maximum, so it is considered as the root node The ‘Y’ descendant has only positive examples and becomes the leaf node with classification ‘Lays Eggs’ Animal Warm-blooded Feathers Fur Swims Lays Eggs Ostrich Yes No Crocodile Raven Albatross Dolphin Koala Feathers Y N [Ostrich, Raven, Albatross] [Crocodile, Dolphin, Koala] Lays Eggs ?

We now repeat the procedure, S: [Crocodile, Dolphin, Koala] S: [1+,2-]
Animal Warm-blooded Feathers Fur Swims Lays Eggs Crocodile No Yes Dolphin Koala We now repeat the procedure, S: [Crocodile, Dolphin, Koala] S: [1+,2-] Entropy(S) = -(1/3)log2(1/3) – (2/3)log2(2/3) =

For attribute ‘Warm-blooded’:
Values(Warm-blooded) : [Yes,No] S = [1Y,2N] SYes = [0Y,2N] E(SYes) = 0 SNo = [1Y,0N] E(SNo) = 0 Gain(S,Warm-blooded) = – [(2/3)*0 + (1/3)*0] = For attribute ‘Fur’: Values(Fur) : [Yes,No] SYes = [0Y,1N] E(SYes) = 0 SNo = [1Y,1N] E(SNo) = 1 Gain(S,Fur) = – [(1/3)*0 + (2/3)*1] = For attribute ‘Swims’: Values(Swims) : [Yes,No] SYes = [1Y,1N] E(SYes) = 1 SNo = [0Y,1N] E(SNo) = 0 Gain(S,Swims) = – [(2/3)*1 + (1/3)*0] = Gain(S,Warm-blooded) is maximum

The final decision tree will be:
Feathers Y N Lays eggs Warm-blooded Lays Eggs Does not lay eggs

Example 2 Factors affecting sunburn Name Hair Height Weight Lotion
Sunburned Sarah Blonde Average Light No Yes Dana Tall Alex Brown Short Annie Emily Red Heavy Pete John Katie

In this case, the final decision tree will be
Hair Blonde Brown Red Sunburned Not Sunburned Lotion N Y Not Sunburned Sunburned

References "Machine Learning", by Tom Mitchell, McGraw-Hill, 1997
"Building Decision Trees with the ID3 Algorithm", by: Andrew Colin, Dr. Dobbs Journal, June 1996 Professor Sin-Min Lee, SJSU.