Efficient Exact Set-Similarity Joins Arvind Arasu Venkatesh Ganti Raghav Kaushik DMX Group, Microsoft Research

Sept. 15, 2006Set-Similarity Joins2 Data Cleaning NameStreetCityStateZip INGRAM MICRO 1600 ST ANDREWS PL SANTA ANA CA92799 GTE CORP 1 STAMFORD FORUM STAMFORDCT LOGISOFT 274 GOODMAN ST N ROCHESTER14607 CIEDC TH ST LINCONLIL92799 INGRAM MCRO 1600 ST ANDREW’S PL SANTA ANA CA92799

Sept. 15, 2006Set-Similarity Joins3 Data Cleaning NameStreetCityStateZip INGRAM MICRO 1600 ST ANDREWS PL SANTA ANA CA92799 GTE CORP 1 STAMFORD FORUM STAMFORDCT LOGISOFT 274 GOODMAN ST N ROCHESTER14607 CIEDC TH ST LINCONLIL92799 INGRAM MCRO 1600 ST ANDREW’S PL SANTA ANA CA92799

Sept. 15, 2006Set-Similarity Joins4 Data Cleaning NameStreetCityStateZip INGRAM MICRO 1600 ST ANDREWS PL SANTA ANA CA92799 GTE CORP 1 STAMFORD FORUM STAMFORDCT LOGISOFT 274 GOODMAN ST N ROCHESTER14607 CIEDC TH ST LINCONL IL92799 INGRAM MCRO 1600 ST ANDREW’S PL SANTA ANA CA92799

Sept. 15, 2006Set-Similarity Joins5 Data Cleaning NameStreetCityStateZip INGRAM MICRO 1600 ST ANDREWS PL SANTA ANA CA92799 GTE CORP 1 STAMFORD FORUM STAMFORDCT LOGISOFT 274 GOODMAN ST N ROCHESTER14607 CIEDC TH ST LINCONL IL92799 INGRAM MCRO 1600 ST ANDREW’S PL SANTA ANA CA92799

Sept. 15, 2006Set-Similarity Joins6 Data Cleaning NameStreetCityStateZip INGRAM MICRO 1600 ST ANDREWS PL SANTA ANA CA92799 GTE CORP 1 STAMFORD FORUM STAMFORDCT06901 LOGISOFT 274 GOODMAN ST N ROCHESTERNY14607 CIEDC TH ST LINCOLN IL92799

Sept. 15, 2006Set-Similarity Joins7 String Similarity Join CITY ALABASTER ALBERTVILLE … … … LINCOLN … … YUCAIPA Reference Table……City………………… …… LINCONL …… …………… ……………

Sept. 15, 2006Set-Similarity Joins8 NameStreetCityStateZip INGRAM MICRO 1600 ST ANDREWS PL SANTA ANA CA92799 GTE CORP 1 STAMFORD FORUM STAMFORDCT LOGISOFT 274 GOODMAN ST N ROCHESTER14607 CIEDC TH ST LINCONLIL92799 INGRAM MCRO 1600 ST ANDREW’S PL SANTA ANA CA92799 String Similarity (Self) Join

Sept. 15, 2006Set-Similarity Joins9 Strings  Sets [CGK ’06] microsoftmcrosoft {mc, cr, ro, os, so, of, ft}{mi, ic, cr, ro, os, so, of, ft} (edit distance ≤ 1) ----> (Δ ≤ 4) 2-grams

mcrosoft … … … … … … … microsoft … … … … … … … SR String Sim Join edit distance ≤ 1 Strings  Sets

mcrosoft … … … … … … … microsoft … … … … … … … Set Sim Join Δ ≤ 4 RS Tokenize Post-Process Strings  Sets

Sept. 15, 2006Set-Similarity Joins12 String  Set: Advantages Generalizes to many string similarity funcs Generalizes to many string similarity funcs Powerful primitive Powerful primitive Sets ≈ Relations Sets ≈ Relations Leverage relational data processing Leverage relational data processing [CGK ‘06] [CGK ‘06]

Sept. 15, 2006Set-Similarity Joins13 Contributions New algorithms for set-similarity joins New algorithms for set-similarity joins Exact answers Exact answers Performance guarantees Performance guarantees Outperform previous exact algorithms Outperform previous exact algorithms Orders of magnitude Orders of magnitude Exact answers are important for operators

Sept. 15, 2006Set-Similarity Joins14 Outline Introduction Introduction Algorithms Algorithms Experiments Experiments Conclusion Conclusion

{ mi, ic, cr, ro, os, so, of, ft } { lo, og, gi, is, so, of, ft } { … } { bo, oe, ei, in, ng }{ mc, cr, ro, os, so, of, ft } { lg, gi, is, so, of, ft } { … } S R

{ mi, ic, cr, ro, os, so, of, ft } { lo, og, gi, is, so, of, ft } { … } { bo, oe, ei, in, ng }{ mc, cr, ro, os, so, of, ft } { lg, gi, is, so, of, ft } { … } S R Intersection size ≥ 5

{ mi, ic, cr, ro, os, so, of, ft } { lo, og, gi, is, so, of, ft } { … } { bo, oe, ei, in, ng }{ mc, cr, ro, os, so, of, ft } { lg, gi, is, so, of, ft } { … } S R Intersection size ≥ 5

{ mi, ic, cr, ro, os, so, of, ft } { lo, og, gi, is, so, of, ft } { … } { bo, oe, ei, in, ng }{ mc, cr, ro, os, so, of, ft } { lg, gi, is, so, of, ft } { … } S R Intersection size ≥ 5

{ mi, ic, cr, ro, os, so, of, ft } { lo, og, gi, is, so, of, ft } { … } { bo, oe, ei, in, ng } { mc, cr, ro, os, so, of, ft } { lg, gi, is, so, of, ft } { … } S R { mc, cr, ro, os, so, of, ft } { mi, ic, cr, ro, os, so, of, ft } Intersection size ≥ 5

{ mi, ic, cr, ro, os, so, of, ft } { lo, og, gi, is, so, of, ft } { … } { bo, oe, ei, in, ng } { mc, cr, ro, os, so, of, ft } { lg, gi, is, so, of, ft } { … } S R { mc, cr, ro, os, so, of, ft } { mi, ic, cr, ro, os, so, of, ft } Intersection size ≥ 5 { lg, gi, is, so, of, ft } { lo, og, gi, is, so, of, ft }

{ … } { bo, oe, ei, in, ng } { … } S R { mc, cr, ro, os, so, of, ft } { mi, ic, cr, ro, os, so, of, ft } Sim ( r i, s j ) ≥ θ { lg, gi, is, so, of, ft } { lo, og, gi, is, so, of, ft } s2s2s2s2 s3s3s3s3 smsmsmsm s1s1s1s1 r2r2r2r2 r3r3r3r3 rnrnrnrn r1r1r1r1

{ … } { bo, oe, ei, in, ng } { … } S R { mc, cr, ro, os, so, of, ft } { mi, ic, cr, ro, os, so, of, ft } Sim ( r i, s j ) ≥ θ { lg, gi, is, so, of, ft } { lo, og, gi, is, so, of, ft } s2s2s2s2 s3s3s3s3 smsmsmsm s1s1s1s1 r2r2r2r2 r3r3r3r3 rnrnrnrn r1r1r1r1 Large

Input: Input: R: r 1, r 2, …, r n (n sets) R: r 1, r 2, …, r n (n sets) S: s 1, s 2, …, s m (m sets) S: s 1, s 2, …, s m (m sets) Output: All pairs (r i, s j ) such that: Output: All pairs (r i, s j ) such that: |r i Δ s j | ≤ k |r i Δ s j | ≤ k Set-Similarity Join: Symmetric Difference ≤ k Running example: k = 4

Sept. 15, 2006Set-Similarity Joins24 Alternate Set Representation s = { 4, 10, 13, 24, 29, 35, 41, 46, 48 }

Sept. 15, 2006Set-Similarity Joins25 Alternate Set Representation s = { 4, 10, 13, 24, 29, 35, 41, 46, 48 } 12550

Sept. 15, 2006Set-Similarity Joins26 Alternate Set Representation s = { 4, 10, 13, 24, 29, 35, 41, 46, 48 } 12550

Sept. 15, 2006Set-Similarity Joins27 Alternate Set Representation s = { 4, 10, 13, 24, 29, 35, 41, 46, 48 } 12550

Sept. 15, 2006Set-Similarity Joins28 Alternate Set Representation s = { 4, 10, 13, 24, 29, 35, 41, 46, 48 } 12550

Sept. 15, 2006Set-Similarity Joins29 Enumeration s r |r Δ s | ≤ 4

Sept. 15, 2006Set-Similarity Joins30 Enumeration s r |r Δ s | ≤ 4

Sept. 15, 2006Set-Similarity Joins31 Enumeration s r |r Δ s | ≤ 4 Errors

Sept. 15, 2006Set-Similarity Joins32 Enumeration s r |r Δ s | ≤ 4

Sept. 15, 2006Set-Similarity Joins33 Enumeration: Signature Generation s,,,,{} Sig (s )

Sept. 15, 2006Set-Similarity Joins34 Enumeration: Signature Generation s,,,,{} Sig (s ) { 0x4f72ba91, 0x29c8af10, 0x594b2c17, 0xa3b0e20f, 0xdd21f32a} Hash32()

Sept. 15, 2006Set-Similarity Joins35 Property of Signatures |r Δ s | ≤ 4 Sig (r ) Sig (s ) ≠ Φ U s r

Sept. 15, 2006Set-Similarity Joins36 Enumeration: Algorithm Generate signatures for each r i, s j Generate signatures for each r i, s j Enumerate (r i, s j ) s.t Sig (r i ) Sig (s j ) ≠ Φ Enumerate (r i, s j ) s.t Sig (r i ) Sig (s j ) ≠ Φ Output those satisfying |r i Δ s j | ≤ 4 Output those satisfying |r i Δ s j | ≤ 4 U

Sept. 15, 2006Set-Similarity Joins37 Enumeration s1s1 s5s5 s2s2 s3s3 s4s4 Sig (s 2 ) Sig (s 5 ) Sig (s 3 ) Sig (s 4 ) U r1r1 r5r5 r2r2 r3r3 r4r4 Sig (s 1 ) Sig (r 2 ) Sig (r 5 ) Sig (r 3 ) Sig (r 4 ) Sig (r 1 ) Sig (r 2 ) Sig (s 1 ) ≠ Φ

Sept. 15, 2006Set-Similarity Joins38 Enumeration s1s1 s5s5 s2s2 s3s3 s4s4 Sig (s 2 ) Sig (s 5 ) Sig (s 3 ) Sig (s 4 ) U r1r1 r5r5 r2r2 r3r3 r4r4 Sig (s 1 ) Sig (r 2 ) Sig (r 5 ) Sig (r 3 ) Sig (r 4 ) Sig (r 1 ) Sig (r 2 ) Sig (s 1 ) ≠ Φ

Sept. 15, 2006Set-Similarity Joins39 Enumeration s1s1 s5s5 s2s2 s3s3 s4s4 Sig (s 2 ) Sig (s 5 ) Sig (s 3 ) Sig (s 4 ) U r1r1 r5r5 r2r2 r3r3 r4r4 Sig (s 1 ) Sig (r 2 ) Sig (r 5 ) Sig (r 3 ) Sig (r 4 ) Sig (r 1 ) Sig (r 2 ) Sig (s 1 ) ≠ Φ Output False positive candidate pairs

S (Id, Elem) R.Sig = S.Sig δ R.Id, S.Id R (Id, Elem) Post-Process each R.Id, S.Id Gen Signatures S’ (Id, Sig)R’ (Id, Sig)

Sept. 15, 2006Set-Similarity Joins41 No False Positive Candidate Pair s r |r Δ s | = 5

Sept. 15, 2006Set-Similarity Joins42 False Positive Candidate Pair s2s2 s1s |r Δ s | = 5

Sept. 15, 2006Set-Similarity Joins43 Enumeration: Performance k = 4

Sept. 15, 2006Set-Similarity Joins44 Enumeration: Performance Ideal Performance k = 4

Sept. 15, 2006Set-Similarity Joins45 Enumeration |r Δ s | ≤ 4 s r

Sept. 15, 2006Set-Similarity Joins46 Enumeration s r |r Δ s | ≤ 4

Sept. 15, 2006Set-Similarity Joins47 Enumeration: Signature Generation s1s

Sept. 15, 2006Set-Similarity Joins48 Enumeration: Signature Generation s1s

Sept. 15, 2006Set-Similarity Joins49 Enumeration: Signature Generation s1s

Sept. 15, 2006Set-Similarity Joins50 Enumeration: Signature Generation s1s

Sept. 15, 2006Set-Similarity Joins51 Enumeration: Signature Generation s1s ( ) 6 2 = 15

Sept. 15, 2006Set-Similarity Joins52 Algorithm Generate signatures for each r i, s j Generate signatures for each r i, s j Enumerate (r i, s j ) s.t Sig (r i ) Sig (s j ) ≠ Φ Enumerate (r i, s j ) s.t Sig (r i ) Sig (s j ) ≠ Φ Output those satisfying |r i Δ s j | ≤ 4 Output those satisfying |r i Δ s j | ≤ 4 U Only the signature function changes

Sept. 15, 2006Set-Similarity Joins53 Enumeration: Performance k = 4

Sept. 15, 2006Set-Similarity Joins54 False Positive Candidate Pair s r |r Δ s | = 5

Sept. 15, 2006Set-Similarity Joins55 Enumeration: Performance k = 4

Sept. 15, 2006Set-Similarity Joins56 Enumeration: Performance k = 4

Sept. 15, 2006Set-Similarity Joins57 PartEnum: Divide and Conquer s1s1 21 k = 4 k 2 = 1 k 1 = 2 Generate signatures using Enumeration

Sept. 15, 2006Set-Similarity Joins58 PartEnum: Asymptotic Performance Theorem: There is an instance of PartEnum such that: Theorem: There is an instance of PartEnum such that: If |r Δ s | > 7.5 k, then r and s do not share a signature with probability 1 – o(1) If |r Δ s | > 7.5 k, then r and s do not share a signature with probability 1 – o(1) The number of signatures per set: O (k 2 ) The number of signatures per set: O (k 2 )

Sept. 15, 2006Set-Similarity Joins59 PartEnum: Summary Set-Similarity Joins with predicate |r Δ s | ≤ k Set-Similarity Joins with predicate |r Δ s | ≤ k Theoretical guarantees Theoretical guarantees First exact algorithm First exact algorithm

Sept. 15, 2006Set-Similarity Joins60 Other results PartEnum extensions: PartEnum extensions: Larger class of set-similarity join predicates Larger class of set-similarity join predicates Jaccard Jaccard Basic idea: reduce to symmetric set difference Basic idea: reduce to symmetric set difference WtEnum class of signature functions: WtEnum class of signature functions: Use frequency of elements Use frequency of elements Weighted set-similarity joins Weighted set-similarity joins

Sept. 15, 2006Set-Similarity Joins61 Outline Introduction Introduction Algorithms Algorithms Experiments Experiments Conclusion Conclusion

S (Id, Elem) R.Sig = S.Sig δ R.Id, S.Id R (Id, Elem) Post-Process each R.Id, S.Id Gen Signatures Implementation DBMS Client + DBMS DBMS Client

Sept. 15, 2006Set-Similarity Joins63 Previous Work Prefix Filtering [CGK ’06] Prefix Filtering [CGK ’06] Exact Exact Locality Sensitive Hashing [IM ’98] Locality Sensitive Hashing [IM ’98] Approximate Approximate False negative rate: 5% False negative rate: 5%

Sept. 15, 2006Set-Similarity Joins64 Data Sets Organization addresses [MS Sales] Organization addresses [MS Sales] Concatenation: Org name, street, city, zip Concatenation: Org name, street, city, zip Input size: 1 million Input size: 1 million Avg. length: 11 words, 58 chars Avg. length: 11 words, 58 chars Tokenization: Words, n-grams Tokenization: Words, n-grams

Sept. 15, 2006Set-Similarity Joins65 Jaccard, 1M, MS Sales

S (Id, Elem) R.Sig = S.Sig δ R.Id, S.Id R (Id, Elem) Post-Process each R.Id, S.Id Gen Signatures Evaluation DBMS DBMS Intermediate Result size Client + DBMS Client

Jaccard, 1M, MS Sales

Sept. 15, 2006Set-Similarity Joins68 Jaccard, Synthetic

Sept. 15, 2006Set-Similarity Joins69 Similar Results for … Other data sets Other data sets DBLP, Synthetic data sets DBLP, Synthetic data sets Other similarity functions Other similarity functions Weighted jaccard Weighted jaccard Edit distance Edit distance

Sept. 15, 2006Set-Similarity Joins70 Conclusion New algorithms for set-similarity joins New algorithms for set-similarity joins Exact Exact Performance guarantees Performance guarantees Outperform previous exact algorithms Outperform previous exact algorithms Search: “data cleaning project”