# Learning Rules from Data

## Presentation on theme: "Learning Rules from Data"— Presentation transcript:

Learning Rules from Data
Olcay Taner Yıldız, Ethem Alpaydın Department of Computer Engineering Boğaziçi University

Rule Induction? Derive meaningful rules from data
Mainly used in classification problems Attribute types (Continuous, Discrete) Name Income Owns a house? Marital status Default Ali 25,000 \$ Yes Married No Veli 18,000 \$ No Married Yes

Rules Disjunction: conjunctions are binded via OR's
Conjunction: propositions are binded via AND's Proposition: relation on an attribute Attribute is Continuous (defines a subinterval) Attribute is Discrete (= one of the values of the attribute)

How to generate rules? Rule Induction Techniques Via Trees
C4.5Rules Directly from Data Ripper

C4.5Rules (Quinlan, 93) Create decision tree using C4.5
Convert the decision tree to a ruleset by writing each path from root to the leaves as a rule

C4.5Rules q1 x1 > q1 x2 > q2 y = 0 y = 1 yes no q2 x2 : savings
x1 : yearly-income q1 x1 > q1 x2 > q2 y = 0 y = 1 yes no OK DEFAULT q2 Rules: IF (x1 > q1) AND (x2 > q2) THEN Y = OK IF (x1 < q1) OR ((x1 > q1) AND (x2 < q2)) THEN Y = DEFAULT

RIPPER (Cohen, 95) Learn rule for each class Two Steps
Objective class is positive, other classes are negative Two Steps Initialization Learn rules one by one Immediately prune learned rule Optimization Since search is greeedy, pass k times over the rules to optimize

RIPPER (Initialization)
Split (pos, neg) into growset and pruneset Rule := grow conjunction with growset Add propositions one by one IF (x1 > q1) AND (x2 > q2) AND (x2 < q3) THEN Y = OK Rule := prune conjunction with pruneset Remove propositions IF (x1 > q1) AND (x2 < q3) THEN Y = OK If MDL < Best MDL + 64 Add conjunction Else Break

Ripper (Optimization)
Repeat K times For each rule IF (x1 > q1) AND (x2 < q3) THEN Y = OK Generate revisionrule by adding propositions IF (x1 > q1) AND (x2 < q3) AND (x1 > q3) THEN Y = OK Generate replacementrule by regrowing IF (x1 > q4) THEN Y = OK Compare current rule with revisionrule and replacementrule Take the best according to MDL

Minimum Description Length
Description Length of a Ruleset Description Length of Rules S = ||k|| + k log2 (n / k) + (n – k) log2 (k / (n – k)) Description Length of Exceptions S = log2 (|D| + 1) + fp (-log2 (e / 2C)) + (C – fp) (-log2 (1 - e / 2C)) + fn (-log2 (fn / 2U)) + (U – fn) (-log2 (1 - fn / 2U))

Ripper* Finding best condition is done by trying all possible split points (time-consuming) Shortcut: Linear Discriminant Analysis Split point is calculated analytically To be more robust Instances further than 3  are removed If number of examples < 20, shortcut not used

Experiments 29 datasets from UCI repository are used
10 fold cross-validation Comparison done using one-sided t test Comparison of three algorithms C4.5Rules, Ripper, Ripper* Comparison based on Error Rate Complexity of the rulesets Learning Time

Error Rate (I) Ripper and its variant have better performance than C4.5Rules Ripper* has similar performance compared to Ripper C4.5Rules has advantage when the number of rules are small (Exhaustive Search)

Error Rate (II) C4.5Rules Ripper Ripper* Total - 4 7 12 2 10 1 5 9

Ruleset Complexity (I)
Ripper and Ripper* produce significantly small number of rules compared to C4.5Rules C4.5Rules starts with an unpruned tree, which is a large amount of rule to start

Ruleset Complexity (II)
C4.5Rules Ripper Ripper* Total - 1 26 10 27 2 3 11

Learning Time (I) Ripper* better than Ripper, which is better than C4.5Rules C4.5Rules O(N3) Ripper O(Nlog2N) Ripper* O(NlogN)

Learning Time (II) C4.5Rules Ripper Ripper* Total - 2 23 25 13 27

Conclusion Comparison of two rule induction algorithms C4.5Rules and Ripper Proposed a shortcut in learning conditions using LDA (Ripper*) Ripper is better than C4.5Rules Ripper* improves learning time of Ripper without decreasing its performance

Similar presentations