1 Mining Associations Apriori Algorithm

2 Computation Model uTypically, data is kept in a flat file rather than a database system. wStored on disk. wStored basket-by-basket. uThe true cost of mining disk- resident data is usually the number of disk I/O’s. uIn practice, association-rule algorithms read the data in passes – all baskets read in turn. uThus, we measure the cost by the number of passes an algorithm takes. Item Basket 1 Basket 2 Basket 3 Etc.

3 Main-Memory Bottleneck uFor many frequent-itemset algorithms, main memory is the critical resource. wAs we read baskets, we need to count something, e.g., occurrences of pairs. wThe number of different things we can count is limited by main memory. wSwapping counts in/out is a disaster.

4 Finding Frequent Pairs uThe hardest problem often turns out to be finding the frequent pairs. uWe’ll concentrate on how to do that, then discuss extensions to finding frequent triples, etc.

5 Naïve Algorithm uRead file once, counting in main memory the occurrences of each pair. wFrom each basket of n items, generate its n (n -1)/2 pairs by two nested loops. uFails if (#items) 2 exceeds main memory. wRemember: #items can be 100K (Wal- Mart) or 10B (Web pages).

6 Details of Main-Memory Counting uTwo approaches: 1.Count all pairs, using a triangular matrix. 2.Keep a table of triples [i, j, c] = the count of the pair of items {i,j } is c. u(1) requires only 4 bytes/pair. wNote: assume integers are 4 bytes. u(2) requires 12 bytes, but only for those pairs with count > 0.

7 4 per pair Method (1) Method (2) 12 per occurring pair

8 Triangular-Matrix Approach – (1) uNumber items 1, 2, …,n uCount {i, j } only if i < j. uKeep pairs in the order w{1,2}, {1,3},…, {1,n}, w{2,3}, {2,4},…,{2,n}, w{3,4},…, {3, n}, w… w{n -1,n}.

9 Triangular-Matrix Approach – (2) uLet n be the number of items. Count for pair {i, j } is at position T(i,j) = (i-1)n - i(i+1)/2 + j {1,2}, {1,3}, {1,4}, {2,3}, {2,4} {3,4} uTotal number of pairs n (n –1)/2; total bytes about 2n 2.

10 Details of Approach #2 uTotal bytes used is about 12p, where p is the number of pairs that actually occur. wBeats triangular matrix if at most 1/3 of possible pairs actually occur. uMay require extra space for retrieval structure, e.g., a hash table.

11 Apriori Algorithm for pairs– (1) uA two-pass approach called a-priori limits the need for main memory. uKey idea: monotonicity : if a set of items appears at least s times, so does every subset. wContrapositive for pairs: if item i does not appear in s baskets, then no pair including i can appear in s baskets.

12 Apriori Algorithm for pairs– (2) uPass 1: Read baskets and count in main memory the occurrences of each item. wRequires only memory proportional to #items. uPass 2: Read baskets again and count in main memory only those pairs whose both elements were found in Pass 1 to be frequent. wRequires memory proportional to square of frequent items only. Item counts Pass 1Pass 2 Frequent items Counts of candidate pairs

13 Detail for A-Priori uYou can use the triangular matrix method with n = number of frequent items. uTrick: number frequent items 1,2,… and keep a table relating new numbers to original item numbers.

14 Frequent Triples, Etc. uFor each k, we construct two sets of k –itemsets: wC k = candidate k - itemsets = those that might be frequent (support > s ) based on information from the pass for k –1. wF k = the set of truly frequent k - itemsets. C1C1 F1F1 C2C2 F2F2 C3C3 Filter Construct First pass Second pass All items All pairs of items from F 1 Count the pairs To be explained Count the items

15 Full Apriori Algorithm wLet k=1 wGenerate frequent itemsets of length 1 wRepeat until no new frequent itemsets are found k=k+1 1.Generate length k candidate itemsets from length k-1 frequent itemsets 2.Prune candidate itemsets containing subsets of length k-1 that are infrequent 3.Count the support of each candidate by scanning the DB and eliminate candidates that are infrequent, leaving only those that are frequent

16 Illustrating Apriori Suppose AB is not in F 2. All these will either not be generated by Step 1 as candidates in C 3, or will be pruned in Step 2.

17 Candidate generation 1.Must ensure that the candidate set is complete. 2.Should not generate the same candidate itemset more than once.

18 Data Set Example s=3

19 F k-1  F 1 Method uExtend each frequent (k - 1)­itemset with a frequent 1-itemset. uIs it complete? uYes, because every frequent k­itemset is composed of wa frequent (k-1)­itemset and wa frequent 1­itemset. uHowever, it doesn’t prevent the same candidate itemset from being generated more than once. wE.g., {Bread, Diapers, Milk} can be generated by merging w{Bread, Diapers} with {Milk}, w{Bread, Milk} with {Diapers}, or w{Diapers, Milk} with {Bread}.

20 Lexicographic Order uKeep frequent itemset sorted in lexicographic order. uEach frequent (k-1)­itemset X is extended with frequent items that are lexicographically larger than the items in X. Example u{Bread, Diapers} can be extended with {Milk} u{Bread, Milk} can’t be extended with {Diapers} u{Diapers, Milk} can’t be extended with {Bread} uWhy is it complete?

21 Prunning uMerging {Beer, Diapers} with {Milk} is unnecessary. Why? uBecause one of its subsets, {Beer, Milk}, is infrequent. uSolution: Prune! uHow?

22 F k-1  F 1 Example {Beer,Diapers,Bread} and {Bread,Milk,Beer} aren't in fact generated if lexicographical ord. is considered.

23 F k-1  F k-1 Method uMerge a pair of frequent (k-1) itemsets only if their first k-2 items are identical. uE.g. frequent itemsets {Bread, Diapers} and {Bread, Milk} are merged to form a candidate 3­itemset {Bread, Diapers, Milk}.

24 F k-1  F k-1 Method uWe don’t merge {Beer, Diapers} with {Diapers, Milk} because the first item in both itemsets is different. But, is this "don't merge" decision Ok? uIndeed, if {Beer, Diapers, Milk} is a viable candidate, it would have been obtained by merging {Beer, Diapers} with {Beer, Milk} instead. Pruning uBecause each candidate is obtained by merging a pair of frequent (k-1)­itemsets, an additional candidate pruning step is needed to ensure that the remaining k-2 subsets of k-1 elements are frequent.

25 F k-1  F k-1 Example

26 Another Example TIDList of item ID’s T1I1, I2, I5 T2I2, I4 T3I2, I3 T4I1, I2, I4 T5I1, I3 T6I2, I3 T7I1, I3 T8I1, I2, I3, I5 T9I1, I2, I3 Min_sup_count = 2 Itemset {I5} {I4} {I3} {I2} {I1} C1 Sup. count Itemset 2{I5} 2{I4} 6{I3} 7{I2} 6{I1} F1

27 Generate C2 from F1  F1 TIDList of item ID’s T1I1, I2, I5 T2I2, I4 T3I2, I3 T4I1, I2, I4 T5I1, I3 T6I2, I3 T7I1, I3 T8I1, I2, I3, I5 T9I1, I2, I3 Min_sup_count = 2 ItemsetSup. count {I1}6 {I2}7 {I3}6 {I4}2 {I5}2 F1 {I4,I5} {I3,I5} {I3,I4} {I2,I5} {I2,I4} Itemset {I2,I3} {I1,I5} {I1,I4} {I1,I3} {I1,I2} C2 ItemsetSup. C {I1,I2}4 {I1,I3}4 {I1,I4}1 {I1,I5}2 {I2,I3}4 {I2,I4}2 {I2,I5}2 {I3,I4}0 {I3,I5}1 {I4,I5}0

28 Generate C3 from F2  F2 TIDList of item ID’s T1I1, I2, I5 T2I2, I4 T3I2, I3 T4I1, I2, I4 T5I1, I3 T6I2, I3 T7I1, I3 T8I1, I2, I3, I5 T9I1, I2, I3 Min_sup_count = 2 ItemsetSup. C {I1,I2}4 {I1,I3}4 {I1,I5}2 {I2,I3}4 {I2,I4}2 {I2,I5}2 F2 Itemset {I1,I2,I3} {I1,I2,I5} {I1,I3,I5} {I2,I3,I4} {I2,I3,I5} {I2,I4,I5} Prune Itemset {I1,I2,I3} {I1,I2,I5} {I1,I3,I5} {I2,I3,I4} {I2,I3,I5} {I2,I4,I5} ItemsetSup. C {I1,I2,I3}2 {I1,I2,I5}2 F3 C3

29 Generate C4 from F3  F3 TIDList of item ID’s T1I1, I2, I5 T2I2, I4 T3I2, I3 T4I1, I2, I4 T5I1, I3 T6I2, I3 T7I1, I3 T8I1, I2, I3, I5 T9I1, I2, I3 Min_sup_count = 2 Itemset {I1,I2,I3,I5} C4 {I1,I2,I3,I5} is pruned because {I2,I3,I5} is infrequent ItemsetSup. C {I1,I2,I3}2 {I1,I2,I5}2 F3