1. Market Baskets Frequent Itemsets A-Priori Algorithm
Association Rules
Market Baskets
Frequent Itemsets
A-Priori Algorithm

2. The Market-Basket Model
A large set of items, e.g., things sold in a supermarket.
A large set of baskets, each of which is a small set of the items, e.g., the things one customer buys on one day.

3. Market-Baskets – (2)
Really a general many-many mapping (association) between two kinds of things.
But we ask about connections among “items,” not “baskets.”
The technology focuses on common events, not rare events (“long tail”).

4. Support
Simplest question: find sets of items that appear “frequently” in the baskets.
Support for itemset I = the number of baskets containing all items in I.
Sometimes given as a percentage.
Given a support threshold s, sets of items that appear in at least s baskets are called frequent itemsets.

5. Example: Frequent Itemsets
Items={milk, coke, pepsi, beer, juice}.
Support = 3 baskets.
B1 = {m, c, b} B2 = {m, p, j}
B3 = {m, b} B4 = {c, j}
B5 = {m, p, b} B6 = {m, c, b, j}
B7 = {c, b, j} B8 = {b, c}
Frequent itemsets: {m}, {c}, {b}, {j},
{m,b}
, {b,c}
, {c,j}.

6. Applications – (1)
Items = products; baskets = sets of products someone bought in one trip to the store.
Example application: given that many people buy beer and diapers together:
Run a sale on diapers; raise price of beer.
Only useful if many buy diapers & beer.

7. Applications – (2)
Baskets = sentences; items = documents containing those sentences.
Items that appear together too often could represent plagiarism.
Notice items do not have to be “in” baskets.

8. Applications – (3) Baskets = Web pages; items = words.
Unusual words appearing together in a large number of documents, e.g., “Brad” and “Angelina,” may indicate an interesting relationship.

9. Aside: Words on the Web Many Web-mining applications involve words.
Cluster pages by their topic, e.g., sports.
Find useful blogs, versus nonsense.
Determine the sentiment (positive or negative) of comments.
Partition pages retrieved from an ambiguous query, e.g., “jaguar.”

10. Words – (2) Very common words are stop words.
They rarely help determine meaning, and they block from view interesting events, so ignore them.
The TF/IDF measure distinguishes “important” words from those that are usually not meaningful.

11. Words – (3) TF/IDF = “term frequency, inverse
document frequency”: relates the number of times a word appears to the number of documents in which it appears.
Low values are words like “also” that appear at random.
High values are words like “computer” that may be the topic of documents in which it appears at all.

12. Scale of the Problem
WalMart sells 100,000 items and can store billions of baskets.
The Web has billions of words and many billions of pages.

13. Association Rules If-then rules about the contents of baskets.
{i1, i2,…,ik} → j means: “if a basket contains all of i1,…,ik then it is likely to contain j.”
Confidence of this association rule is the probability of j given i1,…,ik.

14. Example: Confidence An association rule: {m, b} → c.
B1 = {m, c, b} B2 = {m, p, j}
B3 = {m, b} B4 = {c, j}
B5 = {m, p, b} B6 = {m, c, b, j}
B7 = {c, b, j} B8 = {b, c}
An association rule: {m, b} → c.
Confidence = 2/4 = 50%. + _

15. Finding Association Rules
Question: “find all association rules with support ≥ s and confidence ≥ c .”
Note: “support” of an association rule is the support of the set of items on the left.
Hard part: finding the frequent itemsets.
Note: if {i1, i2,…,ik} → j has high support and confidence, then both {i1, i2,…,ik} and {i1, i2,…,ik ,j } will be “frequent.”

16. Computation Model
Typically, data is kept in flat files rather than in a database system.
Stored on disk.
Stored basket-by-basket.
Expand baskets into pairs, triples, etc. as you read baskets.
Use k nested loops to generate all sets of size k.

17. File Organization Example: items are positive integers, and boundaries
Basket 1
Example: items are
positive integers,
and boundaries
between baskets
are –1.
Item
Item
Item
Item
Basket 2
Item
Item
Item
Item
Basket 3
Item
Item
Etc.

18. Computation Model – (2)
The true cost of mining disk-resident data is usually the number of disk I/O’s.
In practice, association-rule algorithms read the data in passes – all baskets read in turn.
Thus, we measure the cost by the number of passes an algorithm takes.

19. Main-Memory Bottleneck
For many frequent-itemset algorithms, main memory is the critical resource.
As we read baskets, we need to count something, e.g., occurrences of pairs.
The number of different things we can count is limited by main memory.
Swapping counts in/out is a disaster (why?).

20. Finding Frequent Pairs
The hardest problem often turns out to be finding the frequent pairs.
Why? Often frequent pairs are common, frequent triples are rare.
Why? Probability of being frequent drops exponentially with size; number of sets grows more slowly with size.
We’ll concentrate on pairs, then extend to larger sets.

21. Naïve Algorithm
Read file once, counting in main memory the occurrences of each pair.
From each basket of n items, generate its n (n -1)/2 pairs by two nested loops.
Fails if (#items)2 exceeds main memory.
Remember: #items can be 100K (Wal-Mart) or 10B (Web pages).

22. Example: Counting Pairs
Suppose 105 items.
Suppose counts are 4-byte integers.
Number of pairs of items: 105(105-1)/2 = 5*109 (approximately).
Therefore, 2*1010 (20 gigabytes) of main memory needed.

23. Details of Main-Memory Counting
Two approaches:
Count all pairs, using a triangular matrix.
Keep a table of triples [i, j, c] = “the count of the pair of items {i, j } is c.”
(1) requires only 4 bytes/pair.
Note: always assume integers are 4 bytes.
(2) requires 12 bytes, but only for those pairs with count > 0.

24. 4 per pair
12 per
occurring pair
Method (1)
Method (2)

25. Triangular-Matrix Approach – (1)
Number items 1, 2,…
Requires table of size O(n) to convert item names to consecutive integers.
Count {i, j } only if i < j.
Keep pairs in the order {1,2}, {1,3},…, {1,n }, {2,3}, {2,4},…,{2,n }, {3,4},…, {3,n },…{n -1,n }.

26. Triangular-Matrix Approach – (2)
Find pair {i, j } at the position (i –1)(n –i /2) + j – i.
Total number of pairs n (n –1)/2; total bytes about 2n 2.

27. Details of Approach #2
Total bytes used is about 12p, where p is the number of pairs that actually occur.
Beats triangular matrix if at most 1/3 of possible pairs actually occur.
May require extra space for retrieval structure, e.g., a hash table.

28. A-Priori Algorithm for Pairs
A two-pass approach called a-priori limits the need for main memory.
Key idea: monotonicity : if a set of items appears at least s times, so does every subset.
Contrapositive for pairs: if item i does not appear in s baskets, then no pair including i can appear in s baskets.

29. Apriori Algorithm - General
Agrawal & Srikant 1994
Data base D
1-candidates
Freq 1-itemsets
2-candidates
TID
Items 10
a, c, d 20
b, c, e 30
a, b, c, e 40
b, e
Itemset
Sup a 2 b 3 c d 1 e
Itemset
Sup a 2 b 3 c e
Itemset ab ac ae bc be ce
Scan D
Min_sup=2
3-candidates
Freq 2-itemsets
Counting
Scan D
Itemset
bce
Itemset
Sup ac 2 bc be 3 ce
Itemset
Sup ab 1 ac 2 ae bc be 3 ce
Scan D
Freq 3-itemsets
Itemset
Sup
bce 2

30. Important Details of Apriori
How to generate candidates?
Step 1: self-joining Lk
Step 2: pruning
How to count supports of candidates?

31. How to Generate Candidates?
Suppose the items in Lk-1 are listed in an order
Step 1: self-join Lk-1
INSERT INTO Ck
SELECT p.item1, p.item2, …, p.itemk-1, q.itemk-1
FROM Lk-1 p, Lk-1 q
WHERE p.item1=q.item1, …, p.itemk-2=q.itemk-2, p.itemk-1 < q.itemk-1
Step 2: pruning
For each itemset c in Ck do
For each (k-1)-subsets s of c do if (s is not in Lk-1) then delete c from Ck

32. Example of Candidate-generation
L3={abc, abd, acd, ace, bcd}
Self-joining: L3*L3
abcd from abc and abd
acde from acd and ace
Pruning:
acde is removed because ade is not in L3
C4={abcd}

33. How to Count Supports of Candidates?
Why counting supports of candidates a problem?
The total number of candidates can be very huge
One transaction may contain many candidates
Method:
Candidate itemsets are stored in a hash-tree
Leaf node of hash-tree contains a list of itemsets and counts
Interior node contains a hash table
Subset function: finds all the candidates contained in a transaction

34. Apriori: Candidate Generation-and-test
Any subset of a frequent itemset must be also frequent — an anti-monotone property
A transaction containing {beer, diaper, nuts} also contains {beer, diaper}
{beer, diaper, nuts} is frequent  {beer, diaper} must also be frequent
No superset of any infrequent itemset should be generated or tested
Many item combinations can be pruned

35. The Apriori Algorithm Ck: Candidate itemset of size k
Lk : frequent itemset of size k
L1 = {frequent items};
for (k = 1; Lk !=; k++) do
Ck+1 = candidates generated from Lk;
for each transaction t in database do increment the count of all candidates in Ck+1 that are contained in t
Lk+1 = candidates in Ck+1 with min_support
return k Lk;

36. All pairs
of items
from L1
Count
the items
Count
the pairs
To be
explained
All
items
Filter
Filter
Construct
Construct C1 L1 C2 L2 C3
Frequent
items
Frequent
pairs
First
pass
Second
pass

37. Challenges of FPM Challenges Improving Apriori: general ideas
Multiple scans of transaction database
Huge number of candidates
Tedious workload of support counting for candidates
Improving Apriori: general ideas
Reduce number of transaction database scans
Shrink number of candidates
Facilitate support counting of candidates

38. DIC: Reduce Scans
ABCD
Once both A and D are determined frequent, the counting of AD can begin
Once all length-2 subsets of BCD are determined frequent, the counting of BCD can begin
ABC
ABD
ACD
BCD AB AC BC AD BD CD
Transactions
1-itemsets A B C D
2-itemsets
Apriori … {}
Itemset lattice
1-itemsets
2-items
S. Brin R. Motwani, J. Ullman, and S. Tsur, 1997.
DIC
3-items

39. Partition: Scan Database Only Twice
Partition the database into n partitions
Itemset X is frequent  X is frequent in at least one partition
Scan 1: partition database and find local frequent patterns
Scan 2: consolidate global frequent patterns
A. Savasere, E. Omiecinski, and S. Navathe, 1995

40. Sampling for Frequent Patterns
Select a sample of original database, mine frequent patterns within sample using Apriori
Scan database once to verify frequent itemsets found in sample, only borders of closure of frequent patterns are checked
Example: check abcd instead of ab, ac, …, etc.
Scan database again to find missed frequent patterns
H. Toivonen, 1996

41. Bottleneck of Frequent-pattern Mining
Multiple database scans are costly
Mining long patterns needs many passes of scanning and generates lots of candidates
To find frequent itemset i1i2…i100
# of scans: 100
# of Candidates:
Bottleneck: candidate-generation-and-test
Can we avoid candidate generation?

42. Set Enumeration Tree Subsets of I can be enumerated systematically
I={a, b, c, d}  a b c d ab ac ad bc bd cd
abc
abd
acd
bcd
abcd

43. Borders of Frequent Itemsets
Connected
X and Y are frequent and X is an ancestor of Y  all patterns between X and Y are frequent  a b c d ab ac ad bc bd cd
abc
abd
acd
bcd
abcd

44. Projected Databases
To find a child Xy of X, only X-projected database is needed
The sub-database of transactions containing X
Item y is frequent in X-projected database  a b c d ab ac ad bc bd cd
abc
abd
acd
bcd
abcd

45. Tree-Projection Method
Find frequent 2-itemsets
For each frequent 2-itemset xy, form a projected database
The sub-database containing xy
Recursive mining
If x’y’ is frequent in xy-proj db, then xyx’y’ is a frequent pattern

46. Borders and Max-patterns
Max-patterns: borders of frequent patterns
A subset of max-pattern is frequent
A superset of max-pattern is infrequent  a b c d ab ac ad bc bd cd
abc
abd
acd
bcd
abcd

47. MaxMiner: Mining Max-patterns
Tid
Items 10
A,B,C,D,E 20
B,C,D,E, 30
A,C,D,F
1st scan: find frequent items
A, B, C, D, E
2nd scan: find support for
AB, AC, AD, AE, ABCDE
BC, BD, BE, BCDE
CD, CE, CDE, DE,
Since BCDE is a max-pattern, no need to check BCD, BDE, CDE in later scan
Baya’98
Min_sup=2
Potential max-patterns

48. Frequent Closed Patterns
For frequent itemset X, if there exists no item y s.t. every transaction containing X also contains y, then X is a frequent closed pattern
“acdf” is a frequent closed pattern
Concise rep. of freq pats
Reduce # of patterns and rules
N. Pasquier et al. In ICDT’99
Min_sup=2
TID
Items 10
a, c, d, e, f 20
a, b, e 30
c, e, f 40
a, c, d, f 50

49. CLOSET: Mining Frequent Closed Patterns
Flist: list of all freq items in support asc. order
Flist: d-a-f-e-c
Divide search space
Patterns having d
Patterns having d but no a, etc.
Find frequent closed pattern recursively
Every transaction having d also has cfa  cfad is a frequent closed pattern
PHM’00
Min_sup=2
TID
Items 10
a, c, d, e, f 20
a, b, e 30
c, e, f 40
a, c, d, f 50

50. Closed and Max-patterns
Closed pattern mining algorithms can be adapted to mine max-patterns
A max-pattern must be closed
Depth-first search methods have advantages over breadth-first search ones