Ivory : Ivory : Pairwise Document Similarity in Large Collection with MapReduce Tamer Elsayed, Jimmy Lin, and Doug Oard Laboratory for Computational Linguistics and Information Processing (CLIP Lab) UM Institute for Advanced Computer Studies (UMIACS)

Problem ~~~~~~~~~~ ~~~~~~~~~~ Applications:  “more-like-that” queries  Clustering e.g., co-reference resolution

Solutions Trivial  For each pair of vectors Compute the inner product  Loads each vector O(N) times Better  Each term contributes only if appears in

Algorithm Loads each posting once Matrix must fit in memory  Works for small collections Otherwise: disk access optimization

Hadoopify : 2-Step Solution 1)Indexing  one MapRedcue step  term  posting file 2)Pairwise Similarity  another MapRedcue step  term contribution for all possible pairs Generate ½ df*(df-1) intermediate contribution / term

Indexing (A,(d 1,2)) (B,(d 1,1)) (C,(d 1,1)) (B,(d 2,1)) (D,(d 2,2)) (A,(d 3,1)) (B,(d 3,2)) (E,(d 3,1)) (A,[(d 1,2), (d 3,1)]) (B,[(d 1,1), (d 2,1), (d 3,2)]) (C,[(d 1,1)]) (D,[(d 2,2)]) (E,[(d 3,1)]) map map map shuffle reduce reduce reduce reduce reduce (A,[(d 1,2), (d 3,1)]) (B,[(d 1,1), (d 2,1), (d 3,2)]) (C,[(d 1,1)]) (D,[(d 2,2)]) (E,[(d 3,1)]) A A B C B D D A B B E d1d1 d2d2 d3d3

Pairwise Similarity map map map map map (A,[(d 1,2), (d 3,1)]) (B,[(d 1,1), (d 2,1), (d 3,2)]) (C,[(d 1,1)]) (D,[(d 2,2)]) (E,[(d 3,1)]) ((d 1,d 3 ),2) ((d 1,d 2 ),1) ((d 1,d 3 ),2) ((d 2,d 3 ),2) shuffle ((d 1,d 2 ),[1]) ((d 1,d 3 ),[2, 2]) ((d 2,d 3 ),[2]) reduce reduce reduce ((d 1,d 2 ),1) ((d 1,d 3 ),4) ((d 2,d 3 ),2)

Implementation Issues df-cut  Drop common terms Intermediate tuples dominated by very high df terms  efficiency Vs. effectiveness Space saving tricks  Common doc + stripes  Blocking  Compression

Experimental Setup Hadoop Cluster of 19 nodes (w/double processors) Aquaint-2 collection  906K documents Okapi BM25 Subsets of collection

Efficiency (running time) 99% df-cut

Efficiency (disk usage)

Effectiveness (recent)

Conclusion Simple and efficient MapReduce solution  2H (using 38 nodes, 99% df-cut) for ~million-doc collection  Play tricks for I/O bound jobs Effective linear-time-scaling approximation  99.9% df-cut achieves 98% relative accuracy  df-cut controls efficiency vs. effectiveness tradeoff

Future work Bigger collections! More investigation of df-Cut and other techniques Analytical model Compression techniques (e.g., bitwise) More effectiveness experiments  Joint resolution of personal names in  Co-reference resolution of names and organization MapReduce IR research platform  Batch query processing

Thank You!

MapReduce Framework Shuffling: group values by keys mapmapmapmap reducereducereduce input output