1 Verifying and Mining Frequent Patterns from Large Windows ICDE2008 Barzan Mozafari, Hetal Thakkar, Carlo Zaniolo Date: 2008/9/25 Speaker: Li, HueiJyun Advisor: Dr. Koh, JiaLing

2 Outline Introduction Mining Large Sliding Windows Verification Double-Tree Verifier (DTV) Depth-First Verifier (DFV) Hybrid Version of Our Verifiers Experiments Conclusion

3 Introduction Normally, finding new rules requires both machines and domain experts Delays by the mining algorithms in detecting new frequent itemsets are acceptable

4 Introduction Propose an algorithm for incremental mining of frequent itemsets that compares favorably with existing algorithms when real-time response is required The performance of the proposed algorithm improves when small delays are acceptable

5 Introduction The on-line verification of old rules is highly desirable in most application scenarios Propose verifiers for verifying the frequency of previously frequent itemsets over new arriving windows

6 Mining Large Sliding Windows Problem Statement and Notations D ： the dataset to be mined (a window), contains several transactions Count(p, D) ： frequency of an itemset p sup(p, D) ： the support of p Minimum support threshold α σ α (D) ： the set of frequent itemsets in D n = |W| / |S| ： the number of slides in each window

7 Mining Large Sliding Windows * The SWIM Algorithm Sliding Window Incremental Miner (SWIM) always maintains a union of the frequent patterns of all slides in the current window W, called Pattern Tree (PT), which is guaranteed to be a superset of the frequent patterns over W

Mining Large Sliding Windows * The SWIM Algorithm Mine the new slide and add its frequent patterns to PT Uses an auxiliary array, aux_array, stores frequency of a pattern for each window, for which the frequency is not known This counting can either be done eagerly (i.e., immediately) or lazily 8

Mining Large Sliding Windows * The SWIM Algorithm At the end of each slides, SWIM outputs all patterns in PT whose frequency at that time is ≥αn|S| We may miss a few patterns due to lack of knowledge at the time of output, but we will report them as delayed when other slides expires

Mining Large Sliding Windows * The SWIM Algorithm Example 1: assume that our input stream is partitioned into slides S 1, S 2, … and we have 3 slides in each window Consider a pattern p which shows up as frequent in S 4 for the first time p.f i : the frequency of p in the i th slide p.freq: p’s cumulative frequency in the current window

Mining Large Sliding Windows * The SWIM Algorithm W 4 ={S 2, S 3, S 4 } p.freq=p.f 4 p.aux_array= W 5 ={S 3, S 4, S 5 } p.freq=p.f 4 +p.f 5 p.aux_array= W 6 ={S 4, S 5, S 6 } p.freq=p.f 4 +p.f 5 +p.f 6 p.aux_array=

Mining Large Sliding Windows * The SWIM Algorithm 12

Mining Large Sliding Windows * SWIM with Adjusted Delay Bound SWIM can be easily modified to only allow a given maximum delay of L slides (0 ≤ L ≤ n-1) Choosing L = 0 guarantees that all frequent patterns are reported immediately once they become frequent in W Choosing L = n-1 leads to the laziest approach, we wait until a slide expires and then compute the frequency of new patterns and update aux_arrays accordingly

Verification Definition 1: D: be a transactional database P: be a given set of arbitrary patterns min_freq: a given minimum frequency A function f is called verifier if it takes D, P and min_freq as input and for each p P returns one of the following: 1. p’s true frequency in D if it has occurred at least min_freq times or otherwise 2. Reports that it has occurred less than min_freq times

Verification In the special case of min_freq = 0, a verifier simply counts the frequency of all p P In general if min_freq > 0, the verifier can skip any pattern whose frequency will be less than min_freq Verification simply verifies counts for a given set of patterns, i.e., verification does not discover additional patterns 15

Verification * Background 16

Verification * Background Use the following notation to describe the verifiers: u.item: the item represented by node u u.freq: the frequency of u, or NIL when unknown u.children: the set of u’s children head(c): the set of all nodes holding item c 17

Verification * Background Use another data structure called pattern tree A fp-tree Each node represents a unique pattern A verifier algorithm computes the frequency of all patterns in a given pattern tree Initially the frequency of each node in the pattern tree is 0, but after verification it would contain the true frequency of the pattern it represents 18

Verification * Double Tree Verifier (DTV) 19

Verification * Double Tree Verifier (DTV) Theoretically, the running time of DTV could be exponential in the worst case due to exponential nature of frequent pattern mining In practice, number of frequent patterns is small when min_freq is not very small 20

Verification * Double Tree Verifier (DTV) The number of different paths in the execution tree of DTV is bounded by the total number of different patterns in the pattern tree The advantage of DTV increases when the minimum support decreases 21

Verification * Depth-First Verifier (DFV) DFV exploits the following optimizations: Ancestor Failure: If a path in the fp-tree already proved to not contain a prefix of the pattern p, then we know that it does not contain p itself either 22

Verification * Depth-First Verifier (DFV) Smaller Sibiling Equivalence: If a path in the fp-tree has already been marked to (or not to) contain a smaller sibiling of the pattern p, then we know that it does (or does not) contain p itself too Parent Success: If a path in the fp-tree has already been marked to contain the parent pattern of p, then we know that it also contains p 23

24

Verification * Depth-First Verifier (DFV) Definition 2 (smallest decisive ancestor): for a given pattern node u and an fp-tree’s node s, smallest decisive ancestor of s is its lowest ancestor t for which either t.item < u.item or t.item ≠ NIL 25

Verification * Hybrid Version of Our Verifier DTV is faster than DFV when there are many transactions in the fp-tree and many patterns in the pattern tree When our tree is small, DFV is more efficient because conditionalization overhead is high Start with DTV until the conditionalizad tree are “small enough” and after that point switch to DFV 26

Experiments P4 machine running Linux, with 1GB of RAM Algorithm is implemented in C dataset: IBM QUEST data generator Kosarak real-world dataset 27

Experiments * Efficiency of Verification Algorithms 28

Experiments * Efficiency of Verification Algorithms * T20I5D50K 29

Experiments * Efficiency of SWIM Algorithms T20I5D1000Ksupport = 1% 30

Experiments * Efficiency of SWIM Algorithms 31

Conclusion The introduction of a very fast algorithm to verify the frequency of a given set of patterns Further improves the performance by simply allowing a small reporting delay 32