Pairwise Document Similarity in Large Collections with MapReduce Tamer Elsayed, Jimmy Lin, and Douglas W. Oard Association for Computational Linguistics, 2008 May 15, 2014 Kyung-Bin Lim

2 / 19 Outline  Introduction  Methodology  Discussion  Conclusion

3 / 19 Pairwise Similarity of Documents  PubMed – “More like this”  Similar blog posts  Google – Similar pages

4 / 19 Abstract Problem  Applications: – Clustering – “more-like-that” queries ~~~~~~~~~~ ~~~~~~~~~~

5 / 19 Outline  Introduction  Methodology  Results  Conclusion

6 / 19 Trivial Solution  Load each vector O(N) times  O(N 2 ) dot products scalable and efficient solution for large collections Goal

7 / 19 Better Solution  Load weights for each term once  Each term contributes O(df t 2 ) partial scores Each term contributes only if appears in

8 / 19 Better Solution  A term contributes to each pair that contains it  For example, if a term t 1 appears in documents x, y, z :  List of documents that contain a particular term: Inverted Index t 1 appears in x, y, z t1 contributes for pairs: (x, y) (x, z) (y, z)

9 / 19 Algorithm

10 / 19 MapReduce Programming  Framework that supports distributed computing on clusters of computers  Introduced by Google in 2004  Map step  Reduce step  Combine step (Optional)  Applications

11 / 19 MapReduce Model

12 / 19 Computation Decomposition reduce  Load weights for each term once  Each term contributes o(df t 2 ) partial scores Each term contributes only if appears in map

13 / 19 MapReduce Jobs  (1) Inverted Index Computation  (2) Pairwise Similarity

14 / 19 Job1: Inverted Index (A,(d 1,2)) (B,(d 1,1)) (C,(d 1,1)) (B,(d 2,1)) (D,(d 2,2)) (A,(d 3,1)) (B,(d 3,2)) (E,(d 3,1)) (A,[(d 1,2), (d 3,1)]) (B,[(d 1,1), (d 2,1), (d 3,2)]) (C,[(d 1,1)]) (D,[(d 2,2)]) (E,[(d 3,1)]) map map map shuffle reduce reduce reduce reduce reduce (A,[(d 1,2), (d 3,1)]) (B,[(d 1,1), (d 2,1), (d 3,2)]) (C,[(d 1,1)]) (D,[(d 2,2)]) (E,[(d 3,1)]) A A B C B D D A B B E d1d1 d2d2 d3d3

15 / 19 Job2: Pairwise Similarity map map map map map (A,[(d 1,2), (d 3,1)]) (B,[(d 1,1), (d 2,1), (d 3,2)]) (C,[(d 1,1)]) (D,[(d 2,2)]) (E,[(d 3,1)]) ((d 1,d 3 ),2) ((d 1,d 2 ),1) ((d 1,d 3 ),2) ((d 2,d 3 ),2) shuffle ((d 1,d 2 ),[1]) ((d 1,d 3 ),[2,2]) ((d 2,d 3 ),[2]) reduce reduce reduce ((d 1,d 2 ),1) ((d 1,d 3 ),4) ((d 2,d 3 ),2)

16 / 19 Implementation Issues  df-cut – Drop common terms  Intermediate tuples dominated by very high df terms  Implemented 99% cut  efficiency Vs. effectiveness

17 / 19 Outline  Introduction  Methodology  Results  Conclusion

18 / 19 Experimental Setup  Hadoop  Cluster of 19 machines – Each with two processors (single core)  Aquaint-2 collection – 2.5GB of text – 906k documents  Okapi BM25  Subsets of collection

19 / 19 Running Time of Pairwise Similarity Comparisons

20 / 19 Number of Intermediate Pairs

21 / 19 Outline  Introduction  Methodology  Results  Conclusion

22 / 19 Conclusion  Simple and efficient MapReduce solution – 2H for ~million-doc collection  Effective linear-time-scaling approximation – 99.9% df-cut achieves 98% relative accuracy – df-cut controls efficiency vs. effectiveness tradeoff