Rule Induction with Extension Matrices Leslie Damon, based on slides by Yuen F. Helbig Dr. Xindong Wu, 1998

Outline  Extension matrix approach for rule induction  The MFL and MCV optimization problems  The HCV solution  Noise handling and discretization in HCV  Comparison of HCV with ID3-like algorithms including C4.5 and C4.5 rules

Attribute-based induction algorithms Attribute based induction concentrates on symbolic and heuristic computations doesn’t require built in knowledge Best known are the ID3-like algorithms low order polynomial in time and space Alternatively, the extension matrix approach Developed by Hong, et al at Univ. of Illinois in 1985 Uses extension matrix as its mathematical basis

 A positive example is such an example that belongs to a known class, say ‘Play’  All the other examples can be called negative examples Positive and Negative Examples (overcast, mild, high, windy) => Play (rainy, hot, high, windy) => Don’t Play

 Negative example matrix is defined as Negative Example Matrix

 when, v + j k  NEM ij NEM ij when, v + j k  NEM ij The extension matrix (EM) of a positive example against NEM, is defined as dead-element Extension Matrix A dead element cannot be used to distinguish a positive example from negative examples

Example Extension Matrix Negative Extension Matrix (NEM) Positive Example

Example Extension Matrix Extension Matrix (EM) Positive Example

e.g., {X 1  1, X 2  0, X 1  1} and {X 1  1, X 3  1, X 2  0} are paths in the extension matrix above A set of ‘n’ non-dead elements that come from ‘i’ different rows is called a path in an extension matrix Attributes Extension matrix Paths in Extension Matrices

Conjunctive Formulas A path in the EM k of the positive example k against NEM corresponds to a conjunctive formula or cover

A path in the EMD of against NE corresponds to a conjunctive formula or cover, which covers against NE and vice-versa Disjunction Matrix  when, otherwise all of Extension Matrix Disjunction

EMD Example Negative Extension Matrix (NEM)

EMD Example Extension Matrix (EM) Positive Example

EMD Example Positive Example Extension Matrix Disjunction (EMD)

EMD Example Positive Example Extension Matrix Disjunction (EMD)

MFL and MCV (1)  The minimum formula problem (MFL)  Generating a conjunctive formula that covers a positive example or an intersecting group of positive examples against NEM and has the minimum number of different conjunctive selectors  The minimum cover problem (MCV)  Seeking a cover that covers all positive examples in PE against NEM and has the minimum number of conjunctive formulae with each conjunctive formula being as short as possible

MFL and MCV (2)  NP-hard  Two complete algorithms are designed to solve them when each attribute domain D i  {i  1,…,a} satisfies |D i |  2  O(na2 a )for MFL  O(n2 a 4 a  pa 2 4 a )for MCV  When |D i |  2, the domain can be decomposed into several, each having base 2

 HCV is a extension matrix based rule induction algorithm which is  Heuristic  Attribute­based  Noise­tolerant  Divides the positive examples into intersecting groups.  Uses HFL heuristics to find a conjunctive formula which covers each intersecting group.  Low­order polynomial time complexity at induction time What is HCV ?

 HFL finds a heuristic conjunctive formula which corresponds to a path in an extension or disjunction matrix  Consists of 4 strategies, applied in turn  Time complexity of O(na 3 ) What is HFL ?

HFL - Fast Strategy Selector [X 5  {normal, dry-peep}] can be a possible selector, which will cover all 5 rows

HFL - Precedence Selector [X 1  1] and [X 3  1] are two inevitable selectors in the above extension matrix

HFL - Elimination Attribute X 2 can be eliminated by X 3

HFL - Least Frequency Attribute X 1 can be eliminated and there still exists a path

HFL Algorithm (1) Procedure HFL(EM; Hfl) S0: Hfl  {} S1: /* the fast strategy */ Try the fast strategy on all these rows which haven't been covered; If successful, add a corresponding selector to Hfl and return(Hfl) S2: /* the precedence strategy */ Apply the precedence strategy to the uncovered rows; If some inevitable selectors are found, add them to Hfl, label all the rows they cover, and go to S1

HFL Algorithm (2) S3: /* the elimination strategy */ Apply the elimination strategy to those attributes that have neither been selected nor eliminated; If an eliminable selector is found, reset all the elements in the corresponding column with *, and go to S2. S4: /* the least­frequency strategy */ Apply the least­frequency strategy to those attributes which have neither been selected nor eliminated, and find a least­frequency selector; Reset all the elements in the corresponding column with *, and go to S2. Return(Hfl)

HCV Algorithm  HCV: partitions the PEs into intersecting groups calls HFL to find the Hfl for each group builds covering formula by doing a logical OR of the Hfls returns the covering formula Hcv

Complexity of HCV  Worst case time complexity  Space requirement  2na

HCV Example

NEM for Pneumonia

HCV Example EM 1 Positive Example 1

HCV Example EM 2 Positive Example 2

HCV Example EM 3 Positive Example 3

HCV Example EM 4 Positive Example 4

HCV Example EM 5 Positive Example 5

HCV Example EM 1 EM 2

HCV Example EM 1 EM 2 EM 3

HCV Example EM 1 EM 2 EM 3 EM 4

HCV Example EM 1 EM 2 EM 3 EM 4 EM 5

HCV Example HFL Step 1: Fast Strategy HFL Rules = {}

HCV Example HFL Step 2: Precedence HFL Rules = {}

HCV Example HFL Step 3: Elimination HFL Rules = {}

HCV Example HFL Rules = {} HFL Step 4: Least-Frequency

HCV Example HFL Step 4: Least-Frequency HFL Rules = {}

HCV Example HFL Step 2: Precedence HFL Rules = {ESR fast }

HCV Example HFL Step 2: Precedence HFL Rules = {ESR fast }, go to S1

HCV Example HFL Step 1: Fast Strategy HFL Rules = {ESR fast, AUSCULTATION normal }

HCV Example HFL Step 1: Fast Strategy HFL Rules = {ESR fast, AUSCULTATION normal }

HCV Example HCV generated rule C4.5rules generated rule

Example (8)

HCV Noise Handling  Don’t care values are dead elements  Approximate partitioning  partitioning of PE into groups can be approximate rather than strict  Stopping criteria  similar to -c option of C4.5

Real-Valued Attributes  HCV uses the Information Gain Heuristic  Stop splitting criteria  Stop if the information gain on all cut points is the same.  Stop if the number of examples to split is less than a certain number.  Limit the total number of intervals.

Comparison (1) Table 1:Number of rules and conditions using Monk 1, 2 and 3 dataset as training set 1, 2 and 3 respectively

Comparison (2) Table 2: Accuracy

Comparison (3)

Conclusions  Rules generated in HCV take the form of variable-valued logic rules, rather than decision trees  HCV generates very compact rules in low-order polynomial time  Noise handling and discretization  Predictive accuracy comparable to the ID3 family of algorithms viz., C4.5, C4.5rules

a  Number of attributes X a  a th attribute e   Vector of positive examples e –  Vector of negative examples  Value of a th attribute in the k th positive example n  Number of negative examples p  Number of positive examples (r ij ) axb  ij th element of axb matrix A(i,j)  ij th element of matrix A Extension Matrix Terminology