1. Performance and Energy Efficiency Evaluation of Big Data Systems Presented by Yingjie Shi Institute of Computing Technology, CAS 2013-10-31

2. BPOE 2013 | HPCChina 2013 Goals of Big Data Systems Larger GreenerFaster

3. BPOE 2013 | HPCChina 2013 Performance V.S. Energy Efficiency Perfor mance Energy Efficien cy Faster & More Powerful Greener & Cheaper More servers Bigger clusters Powerful processors Sophisticated processing algorithms … Lightweight servers Efficient processors Simpler processing algorithms … Tradeoff Evaluation

4. BPOE 2013 | HPCChina 2013 Evaluation of Performance & Energy Efficiency Tradeoff How to measure? AxPUE: Application Level Metrics for Power Usage Effectiveness in Big Data Systems How to get balance? The Implications from Benchmarking Three Big Data Systems

5. BPOE 2013 | HPCChina 2013 Motivation If you can not measure it, you can not improve it. – Lord Kelvin PUE(Power usage effectiveness): a measure of how efficiently a computer data center uses its power; specifically, how much of the power is actually used by the information technology equipment.

6. BPOE 2013 | HPCChina 2013 PUE & Its Variants MetricTimeOrganizationComputing Formulas PUE2007GreenGrid DCiE2008GreenGrid DCeP2008GreenGrid pPUE2012GreenGrid PUE Scalability 2013GreenGrid

7. BPOE 2013 | HPCChina 2013 Motivation Scenario1 Data Management Researcher An Improved Data Classification Algorithm Does it contribute to greening the data centers? Run the Algorithms on Data Center Compare the PUEs No Obvious Variations! PUE can not measure the effectiveness of any changes made upon the data center infrastructure!

8. BPOE 2013 | HPCChina 2013 Motivation Scenario2 Data Center Administrators Give a budget plan of the data center energy consumption in the next year Estimate the data volume based on the business development How to estimate the energy increasement? PUE provides little reference information for data center planning according to data scale and application complexity

9. BPOE 2013 | HPCChina 2013 Calculation Framework PUE AxPUE

10. BPOE 2013 | HPCChina 2013 Definition - ApPUE ApPUE (Application Performance Power Usage Effectiveness): a metric that measures the power usage effectiveness of IT equipments, specifically, how much of the power entering IT equipments is used to improve the application performance. Computation Formulas: Data processing performance of applications The average rate of IT Equipment Energy consumed

11. BPOE 2013 | HPCChina 2013 Definition - AoPUE AoPUE (Application Overall Power Usage Effectiveness ): a metric that measures the power usage effectiveness of the overall data center system, specifically, how much of the total facility power is used to improve the application performance. Computation Formulas: The average rate of Total Facility Energy Used

12. BPOE 2013 | HPCChina 2013 Acquisition – Application Performance Application Category ExamplesMetric Service ApplicationSearch engine, Ad-hoc queries Number of requests answered in unit time Data Analysis Application Data mining, Reporting, Decision support, Log analysis Volume of data processed in unit time Interactive Real-time Application E-commerce, Profile data management Number of transactions completed in unit time High Performance Computing Scientific ComputingNumber of floating-point operations in unit time

13. BPOE 2013 | HPCChina 2013 Acquisition – Benchmark Requirements of Benchmarks –Provide representative workloads for big data applications –Provide a scalable data generation tool BigDataBench –A big data benchmark suite open-sourced recently and publicly available –All the requirements are well fullfilled

14. BPOE 2013 | HPCChina 2013 Experiment Overview Testbed –Data center of 18 racks,362 servers –Sample 8 servers Workloads Two experiments –Different Applications –Different Implementation Algorithms

15. BPOE 2013 | HPCChina 2013 Experiments on Different Applications BigDataBench SVMSortGrepLinpack 17.211.5269.9179.7

16. BPOE 2013 | HPCChina 2013 Experiments on Different Algorithms Two Implementations for Sort –Several reducers with random sampling partitioning –One reducer without partitioning

17. BPOE 2013 | HPCChina 2013 Conclusions We analyze the requirements of application-level energy effectiveness metrics AxPUE in data centers. We propose two novel application-level metrics ApPUE and AoPUE to measure the energy consumed to improve the application performance. The experiment results show that AxPUE could provide meaningful guidance to data center design and optimization.

18. BPOE 2013 | HPCChina 2013 Evaluation of Performance & Energy Efficiency Tradeoff How to measure? AxPUE: Application Level Metrics for Power Usage Effectiveness in Data Centers How to get balance? The Implications from Benchmarking Three Big Data Systems

19. BPOE 2013 | HPCChina 2013 New Solutions ……

20. BPOE 2013 | HPCChina 2013 Users’ Concerns Diverse big data systems under different applications and data volumes How is the performance? How is the energy consumption? What are the differences between them ? Evaluating three respective big data systems using BigDataBench Comparing two of them from performance and energy efficiency Analyzing the running features of three big data systems

21. BPOE 2013 | HPCChina 2013 Experimental Platforms Xeon (Common processor) Atom ( Low power processor) Tilera (Many core processor) CPU Type Intel Xeon E5310 Intel Atom D510Tilera TilePro36 CPU Core 4 cores @ 1.6GHz 2 cores @ 1.66GHz 36 cores @ 500MHz L1 I/D Cache 32KB24KB16KB/8KB L2 Cache4096KB512KB64KB OoO Execution FPU Connection Mode Buffer Sharing TDP Hyper Threading Xeon E5310 Yes BUSNo80WNo Atom D510 NoYesBUSNo13WYes TilePro36YesNoIMESHYes16WNo Basic Information Brief Comparison Hadoop Cluster Information Xeon VS AtomXeon VS Tilera Master/Slaves1/71/7 and 1/1 Comprison Having the same hardware thread number Having the same core number Hadoop setting Following the guidance on Hadoop official website

22. BPOE 2013 | HPCChina 2013 Benchmark Selection BigDataBench A big data benchmark suite from big data applications Respective applications An innovative data generation tool

23. BPOE 2013 | HPCChina 2013 Metrics Performance: Data processed per second (DPS) Energy Efficiency: Application Performance Power Usage Effectiveness(DPJ) Data Input Size DPS = Run Time Data Input Size DPJ = Energy Consumption

24. BPOE 2013 | HPCChina 2013 XeonAtomTilera DPS DPJ General Observations

25. BPOE 2013 | HPCChina 2013 General Observations Data scale has a significant impact on the performance and energy efficiency of big data systems. The performance and energy efficiency trends of different applications are diverse. XeonAtomTilera

26. BPOE 2013 | HPCChina 2013 Xeon VS Atom – DPS

27. BPOE 2013 | HPCChina 2013 Xeon VS Atom – DPJ

28. BPOE 2013 | HPCChina 2013 Xeon VS Atom – DPS & DPJ 500MB1GB10GB25GB50GB 100G B Sort DPS DPJ 3.67 0.87 4.51 1.08 1.89 0.45 1.54 0.36 1.36 0.32 1.40 0.33 Wordcount DPS DPJ 2.27 0.55 2.38 0.58 2.74 0.61 2.84 0.61 2.82 0.62 2.79 0.60 Grep DPS DPJ 1.83 0.48 1.82 0.46 2.30 0.54 2.79 0.62 2.87 0.63 2.89 0.64 Naïve Bayes DPS DPJ 3.83 0.89 3.89 0.87 4.52 1.01 4.64 0.99 4.54 0.97 4.58 0.90 SVM DPS DPJ 3.19 0.69 3.06 0.64 3.17 0.66 3.14 0.67 Xeon is more powerful than Atom on processing capacity. Atom is more energy –saving than Xeon when dealing with simple computation logic applications.

29. BPOE 2013 | HPCChina 2013 Xeon VS Atom --Speedup Atom doesn’t show energy advantage when dealing with complex application

30. BPOE 2013 | HPCChina 2013 Xeon VS Atom -- Summary Xeon is more powerful than Atom on processing capacity. Atom is energy conservation than Xeon when dealing with applications with simple computation logic. Atom doesn’t show energy advantage when dealing with complex applications.

31. BPOE 2013 | HPCChina 2013 Xeon VS Tilera – DPS

32. BPOE 2013 | HPCChina 2013 Xeon VS Tilera – DPJ

33. BPOE 2013 | HPCChina 2013 Xeon VS Tilera – DPS & DPJ 500MB1GB10GB25GB Sort DPS DPJ 3.67 0.48 3.39 0.45 2.41 0.31 2.60 0.34 Wordcount DPS DPJ 5.19 0.67 5.04 0.65 7.35 0.87 7.78 0.92 Grep DPS DPJ 3.60 0.51 3.52 0.48 7.45 0.94 9.93 1.21 Naïve Bayes DPS DPJ 5.91 0.75 5.78 0.70 7.59 0.89 7.94 0.92 Xeon is more powerful than Tilera on processing capacity Tilera is more energy-saving than Xeon when dealing with the simple computation logic and I/O intensive applications Tilera don’t show energy advantage when dealing with complex applications

34. BPOE 2013 | HPCChina 2013 Xeon VS Tilera The DPS of Xeon The DPS of AtomThe DPS of Tilera

35. BPOE 2013 | HPCChina 2013 Xeon VS Tilera The DPS of Tilera Tilera is more suitable to process I/O intensive applications

36. BPOE 2013 | HPCChina 2013 Xeon VS Tilera -- Summary 36 Xeon is more powerful than Tilera on processing capacity. Tilera is more energy conservation than Xeon when dealing with simple computation logic and I/O intensive applications. Tilera don’t show energy advantage when dealing with complex applications. Tilera is more suitable to process I/O intensive applications.

37. BPOE 2013 | HPCChina 2013 Implications The performance of a big data system is not only related to the hardware itself, but also the application type and data volume of workloads. The weak processors aren’t suitable to deal with complex applications. Even they have lower TDP, they don’t show energy cost advantage.

38. BPOE 2013 | HPCChina 2013 Implications Cont. Xeon generally has better processing capacity accompanied with high energy consumption, especially to some light scale-out applications. Atom and Tilera show energy consumption advantage when dealing with light scale-out applications. Tilera exerts energy advantage on processing I/O intensive application.

39. BPOE 2013 | HPCChina 2013