1 A Characterization of Big Data Benchmarks Wen.Xiong Zhibin Yu, Zhendong Bei, Juanjuan Zhao, Fan Zhang, Yubin Zou, Xue Bai, Ye Li, Chengzhong Xu Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences

2 Agenda Background Motivation Methodology Evaluation Conclusion Future work

24/05/2015 ETI Confidential 3 Background Requirements of a benchmark suite Characteristics of different workload-input pairs Spatio-temporal data in a real world system

24/05/2015 ETI Confidential 4 Background (1/3) Requirements of a benchmark suite –a benchmark suite should contain workloads that represent a wide range of application domains. –workloads in a benchmark suite should be as diverse as possible. –a benchmark suite should not have redundant workloads in itself, keeping simulation or measure time as short as possible.

24/05/2015 ETI Confidential 5 Background (1/3) simulation time between different numbers of workload-input pairs After removing redundancy, it can decrease 30% number of workload-input pairs and %40 simulation time.

24/05/2015 ETI Confidential 6 Background (2/3) Characteristics of different workload-input pairs –Characteristics of workloads as the size of input data set changing Stable Unstable

24/05/2015 ETI Confidential 7 Background (3/3) Spatio-temporal data in Shenzhen Transportation System –GPS trajectory data of taxicabs, taxicabs, 90 millions GPS points per day. –Smart card data in metro transportation system, 15+ millions smart cards, 12+ millions transaction records per day.

24/05/2015 ETI Confidential 8 Background (3/3) (1)2000 square kilometers, 18 millions of people. (2)road network in Shenzhen contains vertices and road segments.

24/05/2015 ETI Confidential 9 Motivation Remove redundancy of a typical benchmark suite Provide a benchmark suite for spatio-temporal data

24/05/2015 ETI Confidential 10 Motivation (1/2) Remove redundancy of a typical benchmark suite –To decrease experiment time of benchmarking the objective system by minimizing the number of typical workload-input pairs.

24/05/2015 ETI Confidential 11 Motivation (2/2) Provide a benchmark suite for spatio-temporal data –Representative workloads in our benchmark suite are as follows: transaction count (hotregion) spatiotemporal origin destination (sztod) map matching hotspot monitoring spatiotemporal secondary sort

24/05/2015 ETI Confidential 12 Methodology Typical MapReduce-based workloads Micro architecture level metrics Principal component analysis (PCA) Hierarchical clustering and K-means clustering

24/05/2015 ETI Confidential 13 Methodology Typical MapReduce-based workloads (1/2): indexworkloadsource 1sortHiBench 2wordcountHiBench 3terasortHiBench 4bayesHiBench 5K-meansHiBench 6Nutch indexingHiBench 7pagerankHiBench 8hive-jionHiBench 9Hive-aggregateHiBench 10grepDCBench 11svmDCBench

24/05/2015 ETI Confidential 14 Methodology Typical MapReduce-based workloads (2/2): indexworkloadsource 12ibcfDCBench 13fpgDCBench 14hmmDCBench 15sztodour internal program for trajectory data 16hotregionour internal program for trajectory data

24/05/2015 ETI Confidential 15 Methodology Micro architecture level metrics are as follows: –Instruction per cycle (IPC) –L1 instruction cache miss ratio –L2 instruction cache miss ratio –Last level cache miss ratio –Branch prediction per instruction –Branch miss prediction per instruction –Off-chip bandwidth utilization

24/05/2015 ETI Confidential 16 Methodology Principal Component Analysis: –It can reduce program characteristics while controlling the amount of information that is thrown away.

24/05/2015 ETI Confidential 17 Methodology Hierarchical clustering –Hierarchical clustering is a "bottom up" approach: each observation starts in its own cluster, and workload-input pairs of clusters are merged as one moves up the hierarchy. It is useful in simultaneously looking at multiple clustering possibilities, and we can use a dendrogram for selecting desired number of clusters. K-means clustering –K-means clustering aims to partition n workloads-input pairs into k clusters in which each workload-input pair belongs to the cluster with the nearest mean, where K is a value specified by user.

24/05/2015 ETI Confidential 18 Evaluation (instruction per cycle) 24/05/2015 ETI Confidential 18 The IPC of these sixteen workloads are range from 0.72 to 0.96, with an average value of Wordcount has the lowest IPC value and hotregion has highest value among these workloads.

24/05/2015 ETI Confidential 19 Evaluation (L1 ICache miss ratio) 24/05/2015 ETI Confidential 19 The cache miss ratios of these typical workloads are range from 3.9% to 19.8%, with an average value of 8.9%. svm has the lowest L1 instruction cache miss ratio and hive-aggre has the highest L1 instruction cache miss ratio.

24/05/2015 ETI Confidential 20 Evaluation (L2 ICache miss ratio) 24/05/2015 ETI Confidential 20 The cache misses value of these workloads are range from 23.7% to 64.9%. On average, workloads from DCBench in right side have larger L2 instruction miss rate then workloads from HiBench in the left side. Overall, the L2 cache is ineffective in our experiment platform.

24/05/2015 ETI Confidential 21 Evaluation (branch prediction per instruction ) 24/05/2015 ETI Confidential 21 These values are range from 0.18 to 0.23, with an average value of Hotregion has the lowest value of branch prediction per instruction while nutchindexing has the highest value of branch prediction per instruction.

24/05/2015 ETI Confidential 22 Evaluation (branch missprediction ratio ) 24/05/2015 ETI Confidential 22 These ratios are range from 1.5% to 5.6%, with an average value of 2.7%. Pagerank has the lowest branch miss prediction ratio while nutch indexing has the highest branch miss prediction ratio. The results show that the branch predictor of our processor matches these typical MapReduce based applications.

24/05/2015 ETI Confidential 23 Evaluation (off-chip bandwidth utilization) 24/05/2015 ETI Confidential 23 Among these workloads we evaluated, terasort is the only one that has the highest utilization ratio with a value of 14%. Overall, in our experiment platform, processors significantly over-provision off-chip bandwidth for these typical workloads.

24/05/2015 ETI Confidential 24 Evaluation (Hierarchical clustering )

24/05/2015 ETI Confidential 25 Evaluation (Hierarchical clustering ) (1)strong cluster, three workload-input pairs of same workload clustered together. (2)weak cluster, two workload-input pairs of same workload clustered together. (3)non cluster, no workload-input pairs of same workload clustered together. index cluster typeworkloads 1strong clusterwordcount, sort, terasort 2weak clustersztod, hotregion 3non clustersvm, ibcf

24/05/2015 ETI Confidential 26 Evaluation(K-means clustering) Seclecting 8 workload-input pairs via K-means clustering clusterworkloadsrepresentative 1sztod-98G,hotregion-17G, hmm-16Ghmm-16G 2fpg, ibcf-2Gfpg 3sztod-24G,sztod-49Gsztod-49G 4wordcount-15G,wordcount-30G, wordcount-60G, svm-20G wordcount-30G 5nutchindexing 6hotregion-35G, hotregion-70G, bayes, hive- aggre hotregion-35G 7sort-15G, sort-30G, sort-60G, terasort-25G, terasort-50G, terasort-100G, hive-join, pagerank Sort-60G 8kmeans

24/05/2015 ETI Confidential 27 Evaluation(K-means clustering) sort-60G can be taken as the representative workload-input pair of its group including eight members.

24/05/2015 ETI Confidential 28 Conclusion Redundancy exists in these pioneering benchmark suites –Such as sort and terasort. The workload behavior of trajectory data analysis applications is dramatically affected by their input data sets.

24/05/2015 ETI Confidential 29 Future work Conduct similarity analysis in workload-input pairs at a larger scale. –More metrics and larger input size Fully implement a big data benchmark suite for spatio-temporal data –Data model, data generator and typical workload-input pairs.

Thank You !!! 30