Presentation is loading. Please wait.

Presentation is loading. Please wait.

Date : 2016/08/09 Advisor : Jia-ling Koh Speaker : Yi-Yui Lee

Published byBrett Chapman Modified over 6 years ago

Similar presentations

Presentation on theme: "Date : 2016/08/09 Advisor : Jia-ling Koh Speaker : Yi-Yui Lee"— Presentation transcript:

1 Date : 2016/08/09 Advisor : Jia-ling Koh Speaker : Yi-Yui Lee
MapReduce Approach for Efficient Computation of All-Pairs Similarity Search Date : 2016/08/09 Advisor : Jia-ling Koh Speaker : Yi-Yui Lee

2 Outline Introduction Related Work Approach Experiment Conclusion
Motivation Purpose Related Work Approach Experiment Conclusion Future work

3 Introduction It is very useful to find the similarity of two data
This problem is very common d2 d3 d4 query d1 similarity computing

4 Similarity(d1,d2)=Cosin(d1,d2)
Introduction How to compute the similarity of two documents d1 Content: ABCGH d2 Content: ACDEF Similarity(d1,d2)=Cosin(d1,d2) =1*1 +1*0 +1*1 +0*1 +0*1 +0*1 +1*0 +1*0 vector A B C D E F G H d1 1 d2 |d1||d2| If Cosin(d1,d2)>= threshold t that d1,d2 is similar

5 O(kn2) high time complexity
Motivation All Pairs Similarity Search(APSS) Find all similarity pair in set d1 Need to consider all different pair as candidate in document set d2 d3 d4 d5 O(kn2) high time complexity d6

6 Purpose Find some approach to compute all similar pairs efficiently
Reduce the response time Increase efficient of similarity computation Data Candidate pair selection Result computation Reduce response time Reduce the number of candidate pairs Parallel processing

7 Outline Introduction Related Work Approach Experiment Conclusion
Grouping Inverted list index Approach Experiment Conclusion Future work

8 Grouping Optimizing Parallel Algorithms for All Pairs Similarity Search. WSDM’13 1-norm base grouping strike out group-pairs which can’t generate similar-pairs Similar disimilar d1 d2 d4 d3 G1 G2

9 d’ Similarity upper bound < t
Inverted list index Scaling Up All Pairs Similarity Search. WWW 2007 Not required computing all dimensions Increase efficiency in Result computation step d=d’+d’’ d’=[f1, f2, f3, …fk, 0, 0, 0,…0] d’ Similarity upper bound < t d’’=[0, 0, 0, …fk+1, fk+2,… , fm]

10 Related Work Grouping Inverted list index
Estimate similarity upper bound between group and data Not efficient to decrease candidate Not balance of each group Inverted list index Computing similarity between data and data Architecture not fit for parallel processing

11 Outline Introduction Related Work Approach Experiment Conclusion
Future work

12 Framework Data Grouping Find similar group-pairs
Generate inter-group candidate pairs Verify inter-group similar pairs Find intra-group similar pairs

13 grouping Purpose grouping data with dimension of maximum value Data
Find similar group-pairs Generate inter-group candidate pairs Verify inter-group similar pairs grouping data with dimension of maximum value f1 f2 f3 f4 d1 1 d2 d3 0.45 0.9 d4 0.65 0.76 d5 0.6 0.52 d6 0.5 dimension“f2” dimension“f3” dimension“f1”

14 Find similar group-pairs
computing leader of groups Data Grouping Find similar group-pairs Generate inter-group candidate pairs Verify inter-group similar pairs 1.computing leader of each group 2.find candidate group-pairs Leader(G)=[a1,a2,a3….an] ai=maximum value of feature “i” in G f1 f2 F3 F4 d5 0.6 0.52 d6 0.5 Leader Leader

15 Find similar group-pairs
find candidate group-pairs which maintains similar-pairs possibility(threshold 0.8) Scaling Up All Pairs Similarity Search. WWW 2007 construct inverted index 1.Max in feature A:0.65 2.0.6*0.65<0.8 3. 1.Max in feature B:1 4.0.6* *1>0.8 f1 f2 f3 f4 Leader(G1) max in”f1” 0.6 0.5 0.52 Leader(G2) max in”f3” 0.45 1 Leader(G3) max in”f2” 0.65 vector Index Leader G1 {(f2,0.6),(f3,0.5),(f4,0.52)} Leader G2 {(f3,1)} Leader G3 {(f2,1)} vector Index Leader G1 {(f2,0.6),(f3,0.5),(f4,0.52)}

16 Find similar group-pairs
vector Index Leader G1 f2→f3→f4 Leader G2 f3 Leader G3 f2 vector Index f1 {} f2 {(G1,0.6),(G3,1)} f3 {(G1,0.5),(G2,1)} f4 {(G1,0.52)} Sim’_U(G2,G3)=0 Sim’_U(G1,G2)=0.5*1>0 f1 f2 f3 f4 Leader(G1) max in”f1” 0.6 0.5 0.52 Leader(G2) max in”f3” 0.45 1 f1 f2 f3 f4 Leader(G1) max in”f1” 0.45 1 Leader(G2) max in”f3” 0.65 Vector Index Leader G1 {(f2,0.6),(f3,0.5),(f4,0.52)} Leader G2 {(f3,1)} Leader G3 {(f2,1)} Sim_U(G1,G2)= Sim’_U(G1,G2)+0.6*0.45=0.77 G1,G2 are disimilar

17 Generate inter-group candidate pairs
f1 f2 f3 f4 Leader(G1) max in”f1” 0.6 0.5 0.52 Leader(G2) max in”f3” 0.45 1 Leader(G3) max in”f2” 0.65 Data Grouping Find similar group-pairs Generate inter-group candidate pairs Verify inter-group similar pairs Sim_U(G1,G2)=0.77 Sim_U(G1,G3)=0.99 Sim_U(G1,G2)=0.29 find candidate-pairs in each candidate group-pair d5 d6 G1 d1 d4 G3

18 Candidate pair computation
Optimizing Parallel Algorithms for All Pairs Similarity Search. WSDM’13 f1 f2 f3 f4 d1 1 d4 0.65 0.76 d5 0.6 0.52 d6 0.5 G1 G3 vector 1-norm d6 2 d5 1.72 vector 1-norm d4 1.41 d1 1 Max=0.6 Max=1 Sim(di ,dj) ≤min(maxw(di)*||dj||1, maxw(dj)*||di||1) Sim(Gi ,dj) ≤ maxw(dj)*max (||Gi||1) data in G1→1-norm length>0.8/1 Sim(di∈g3, g1)>t，則||di∈g3||1>t/maxw(G1) data in G3→1-norm length>0.8/0.6=1.33

19 Verify inter-group similar pairs
computing similarity for each candidate Use brute force to get similar pairs Data Grouping Find similar group-pairs Generate inter-group candidate pairs Verify inter-group similar pairs d1 d2 d3 d4 d5 d6

20 Find intra-group similar pairs
The same with Find similar group-pairs Data Grouping Find similar group-pairs Generate inter-group candidate pairs Verify inter-group similar pairs Find intra-group similar pairs V1 V2 V3 G1

21 Group size change the number of group increase the number of group
need to be considered leader variable of group increase the number of group cutting new group form existing group decrease the number of group merge group

22 Increase the number of group
Find the group which has the most number of elements Cut half element as new group G1 f1 f2 f3 f4 1-norm d5 0.6 1.72 d6 0.5 2 d5 0.6 1.72 d6 0.5 2

23 Decrease the number of group
Find the group which has least elements Find another group which the number of element fewer then average G1 f1 f2 f3 f4 Leader 0.6 0.5 0.52 merge G2 f1 f2 f3 f4 Leader 0.45 1 Variable=Max(| leader(g2)[f1] - leader(g1)[f1]| , | leader(g1)[f3] - leader(g2)[f3]|)=0.5 G3 f1 f2 f3 f4 Leader 0.65 1 Variable=Max(| leader(g1)[f1] - leader(g3)[f1]| , | leader(g1)[f2] - leader(g3)[f2]|)=0.4

24 MapReduce Approach Computation independent Lots of Computation Data
Grouping Find similar group-pairs Generate inter-group candidate pairs Verify inter-group similar pairs Find intra-group similar pairs

25 Inverted index (Building index)
mapper d3(0.45, 0.9, 0, 0) d2(0, 1, 0, 0) <f1, 0.45_d3> <f2, 0.9_d3> <f2, 1_d2> reducer <f2, 0.9_d3: 1_d2 > <f1, 0.45_d3>

26 Inverted index Similarity computation
<f2, 0.9_d3: 1_d2 > <f1, 0.45_d3:0_d2> mapper <f2, 0.9_d3: 1_d2 > <f1, 0.45_d3> <d2d3, 0.9*1> reducer similarity(d2,d3)=0.9

27 Verify inter-group similar pairs
d4d5:f1=0.65,f3=0.76,f1=0.6,f3=0.6,f4=0.52 (d4, d5) d4 0.65 0.76 d5 0.6 0.52 mapper <d4d5:f1, 0.65> <d4d5:f3, 0.76> <d4d5:f1, 0.6> <d4d5:f3, 0.6> <d4d5:f4, 0.6> combiner <d4d5, 0.6*0.76> <d4d5, 0.6*0.65> reducer similarity(d4,d5)=0.6* *0.65

28 Outline Introduction Related Work Approach Experiment Conclusion
Future work

29 Dataset Yahoo! Answer dataset 1 Dataset 2 7800 documents
11800 dimension 299 similar pairs Dataset 2 49885 documents 30000 dimension 5047 similar pairs

30 Experiment Part1 pruning effect of each step Part2 response time

31 Experiment Client CPU 2.4GHZ*2 Memory 4GB OS Windows 64 bit Master
單機實驗環境介紹 CPU 3.4G * 4 Memory 8G OS Windows 64 bit 企業版 Hadoop實驗環境介紹 Client CPU 2.4GHZ*2 Memory 4GB OS Windows 64 bit Master 3.4G * 2 16G Ubuntu 64 bit Slave01 Slave02 3.2G * 2 Data Grouping Find similar group-pairs Generate inter-group candidate pairs Verify inter-group similar pairs Find intra-group similar pairs

32 Experiment 1-1 dataset1 Compare the percentage of similar data pairs discovered from intra-segment data pairs

33 Experiment 1-2 dataset1 Compare the percentage of pruned dissimilar segment pairs among all segment pairs

34 Experiment 1-3 dataset1 Compare the percentage of pruned data pairs by the partitioning methods combined with the pruning strategies

35 Experiment 2-1 dataset1 Compare the response time of algorithms in the centralized environment by varying the number of segments

36 Experiment 2-2 dataset2 Compare the response time of SMD and IL in the parallelized environment by varying the number of data

37 Outline Introduction Related Work Approach Experiment Conclusion
Future work

38 Conclusion We propose a maximum base grouping
Use leader vector to computation group-pair similarity Combine MapReduce Parallel Processing Efficient computation of All-Pairs Similarity Search (APSS)

39 Outline Introduction Related Work Approach Experiment Conclusion
Future work

40 Future work The proposed partitioning method can combine with a load assignment policy to achieve further load balance in a parallel environment Extend the proposed partitioning method to support incremental all pairs similarity search efficiently.

Download ppt "Date : 2016/08/09 Advisor : Jia-ling Koh Speaker : Yi-Yui Lee"

Similar presentations

Date: 2014/05/06 Author: Michael Schuhmacher, Simon Paolo Ponzetto Source: WSDM’14 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Knowledge-based Graph Document.

Date: 2014/05/06 Author: Michael Schuhmacher, Simon Paolo Ponzetto Source: WSDM’14 Advisor: Jia-ling Koh Speaker: Chen-Yu Huang Knowledge-based Graph Document.
Sumblr: Continuous Summarization of Evolving Tweet Streams

Sumblr: Continuous Summarization of Evolving Tweet Streams
Diversity Maximization Under Matroid Constraints Date : 2013/11/06 Source : KDD’13 Authors : Zeinab Abbassi, Vahab S. Mirrokni, Mayur Thakur Advisor :

Diversity Maximization Under Matroid Constraints Date : 2013/11/06 Source : KDD’13 Authors : Zeinab Abbassi, Vahab S. Mirrokni, Mayur Thakur Advisor :
Reference-based Indexing of Sequence Databases Jayendra Venkateswaran, Deepak Lachwani, Tamer Kahveci, Christopher Jermaine University of Florida-Gainesville.

Reference-based Indexing of Sequence Databases Jayendra Venkateswaran, Deepak Lachwani, Tamer Kahveci, Christopher Jermaine University of Florida-Gainesville.
TI: An Efficient Indexing Mechanism for Real-Time Search on Tweets Chun Chen 1, Feng Li 2, Beng Chin Ooi 2, and Sai Wu 2 1 Zhejiang University, 2 National.

TI: An Efficient Indexing Mechanism for Real-Time Search on Tweets Chun Chen 1, Feng Li 2, Beng Chin Ooi 2, and Sai Wu 2 1 Zhejiang University, 2 National.
Linking Named Entity in Tweets with Knowledge Base via User Interest Modeling Date : 2014/01/22 Author : Wei Shen, Jianyong Wang, Ping Luo, Min Wang Source.

Linking Named Entity in Tweets with Knowledge Base via User Interest Modeling Date : 2014/01/22 Author : Wei Shen, Jianyong Wang, Ping Luo, Min Wang Source.
Yasuhiro Fujiwara (NTT Cyber Space Labs)

Yasuhiro Fujiwara (NTT Cyber Space Labs)
Distributed Approximate Spectral Clustering for Large- Scale Datasets FEI GAO, WAEL ABD-ALMAGEED, MOHAMED HEFEEDA PRESENTED BY : BITA KAZEMI ZAHRANI 1.

Distributed Approximate Spectral Clustering for Large- Scale Datasets FEI GAO, WAEL ABD-ALMAGEED, MOHAMED HEFEEDA PRESENTED BY : BITA KAZEMI ZAHRANI 1.
An Efficient Multi-Dimensional Index for Cloud Data Management Xiangyu Zhang Jing Ai Zhongyuan Wang Jiaheng Lu Xiaofeng Meng School of Information Renmin.

An Efficient Multi-Dimensional Index for Cloud Data Management Xiangyu Zhang Jing Ai Zhongyuan Wang Jiaheng Lu Xiaofeng Meng School of Information Renmin.
Top-k Set Similarity Joins Chuan Xiao, Wei Wang, Xuemin Lin and Haichuan Shang University of New South Wales and NICTA.

Top-k Set Similarity Joins Chuan Xiao, Wei Wang, Xuemin Lin and Haichuan Shang University of New South Wales and NICTA.
Query Processing of Massive Trajectory Data based on MapReduce Qiang Ma, Bin Yang (Fudan University) Weining Qian, Aoying Zhou (ECNU) Presented By: Xin.

Query Processing of Massive Trajectory Data based on MapReduce Qiang Ma, Bin Yang (Fudan University) Weining Qian, Aoying Zhou (ECNU) Presented By: Xin.
1 NNH: Improving Performance of Nearest- Neighbor Searches Using Histograms Liang Jin (UC Irvine) Nick Koudas (AT&T Labs Research) Chen Li (UC Irvine)

1 NNH: Improving Performance of Nearest- Neighbor Searches Using Histograms Liang Jin (UC Irvine) Nick Koudas (AT&T Labs Research) Chen Li (UC Irvine)
1 Adaptive slice-level parallelism for H.264/AVC encoding using pre macroblock mode selection Bongsoo Jung, Byeungwoo Jeon Journal of Visual Communication.

1 Adaptive slice-level parallelism for H.264/AVC encoding using pre macroblock mode selection Bongsoo Jung, Byeungwoo Jeon Journal of Visual Communication.
Parallel K-Means Clustering Based on MapReduce The Key Laboratory of Intelligent Information Processing, Chinese Academy of Sciences Weizhong Zhao, Huifang.

Parallel K-Means Clustering Based on MapReduce The Key Laboratory of Intelligent Information Processing, Chinese Academy of Sciences Weizhong Zhao, Huifang.
Cost-Based Variable-Length-Gram Selection for String Collections to Support Approximate Queries Efficiently Xiaochun Yang, Bin Wang Chen Li Northeastern.

Cost-Based Variable-Length-Gram Selection for String Collections to Support Approximate Queries Efficiently Xiaochun Yang, Bin Wang Chen Li Northeastern.
Large-Scale Content-Based Image Retrieval Project Presentation CMPT 880: Large Scale Multimedia Systems and Cloud Computing Under supervision of Dr. Mohamed.

Large-Scale Content-Based Image Retrieval Project Presentation CMPT 880: Large Scale Multimedia Systems and Cloud Computing Under supervision of Dr. Mohamed.
Jeffrey D. Ullman Stanford University.  Mining of Massive Datasets, J. Leskovec, A. Rajaraman, J. D. Ullman.  Available for free download at i.stanford.edu/~ullman/mmds.html.

Jeffrey D. Ullman Stanford University.  Mining of Massive Datasets, J. Leskovec, A. Rajaraman, J. D. Ullman.  Available for free download at i.stanford.edu/~ullman/mmds.html.
Graph-based consensus clustering for class discovery from gene expression data Zhiwen Yum, Hau-San Wong and Hongqiang Wang Bioinformatics, 2007.

Graph-based consensus clustering for class discovery from gene expression data Zhiwen Yum, Hau-San Wong and Hongqiang Wang Bioinformatics, 2007.
Design Patterns for Efficient Graph Algorithms in MapReduce Jimmy Lin and Michael Schatz University of Maryland MLG, 2010 23 January, 2014 Jaehwan Lee.

Design Patterns for Efficient Graph Algorithms in MapReduce Jimmy Lin and Michael Schatz University of Maryland MLG, January, 2014 Jaehwan Lee.
Performance Guarantees for Distributed Reachability Queries Wenfei Fan 1,2 Xin Wang 1 Yinghui Wu 1,3 1 University of Edinburgh 2 Harbin Institute of Technology.

Performance Guarantees for Distributed Reachability Queries Wenfei Fan 1,2 Xin Wang 1 Yinghui Wu 1,3 1 University of Edinburgh 2 Harbin Institute of Technology.

Similar presentations

Ads by Google