1. Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic Mohammad Hammoud, M. Suhail Rehman, and Majd F. Sakr 1

2. Hadoop MapReduce MapReduce is now a pervasive analytics engine on the cloud Hadoop is an open source implementation of MapReduce Hadoop MapReduce incorporates two phases, Map and Reduce phases, which encompass multiple map and reduce tasks Map Task Reduce Task Partition To HDFS Dataset HDFS HDFS BLK Map Phase Shuffle Stage Merge Stage Reduce Stage Reduce Phase 2 Partition

3. Task Scheduling in Hadoop A golden principle adopted by Hadoop is: “Moving computation towards data is cheaper than moving data towards computation” Hadoop applies this principle to Map task scheduling but not to Reduce task scheduling With reduce task scheduling, once a slave (or a TaskTracker- TT) polls for a reduce task, R, at the master node (or the JobTracker- JT), JT assigns TT any R CS RS1 TT1 RS2 TT2 TT3 TT4 TT5 JT Request Reduce Task R Assign R to TT1 Shuffle Partitions +1 Total Network Distance (TND) = 4 CS= Core Switch & RS = Rack Switch A locality problem, where R is scheduled at TT1 while its partitions exist at TT4 3

4. Data Locality: A Working Example TT1 and TT2 are feeding nodes of a reduce task R Every TT is requesting R from JT JT can assign R to any TT CS TT1 TND R = 8 CASE-I Hadoop does not distinguish between different cases to choose the one that provides best locality (i.e., CASE-IV or CASE-V) RS1 TT2 TT3 TT4 RS2 TT5 JT CS TT1 TND R = 4 CASE-III RS1 TT2 TT3 TT4 RS2 TT5 JT CS TT1 TND R = 2 CASE-IV RS1 TT2 TT3 TT4 RS2 TT5 JT CS TT1 TND R = 2 CASE-V RS1 TT2 TT3 TT4 RS2 TT5 JT CS TT1 TND R = 8 CASE-II RS1 TT2 TT3 TT4 RS2 TT5 JT 4

5. Partitioning Skew in MapReduce Existing Hadoop’s reduce task scheduler is not only locality unaware, but also partitioning skew unaware Partitioning skew refers to the significant variance in intermediate key’s frequencies and their distribution across different data nodes Partitioning skew has been reported to exist in many scientific applications including feature extraction and bioinformatics, among others Partitioning skew causes shuffle skew where some reduce tasks receive more data than others WordCountSortK-Means 5

6. Partitioning Skew: A Working Example TT1 and TT2 are feeding nodes of a reduce task R TT1 and TT2 are requesting R from JT R’s partitions at TT1 and TT2 are of sizes 100MB and 20MB Amount of data shuffled = 100MB Amount of data shuffled = 20MB Hadoop does not consider partitioning skew exhibited by some MapReduce applications CS TT1 TND R = 2 CASE-IV RS1 TT2 TT3 TT4 RS2 TT5 JT CS TT1 TND R = 2 CASE-V RS1 TT2 TT3 TT4 RS2 TT5 JT 6

7. Our Work We explore the locality and the partitioning skew problems present in the current Hadoop implementation We propose Center-of-Gravity Reduce Scheduler (CoGRS), a locality-aware skew-aware reduce task scheduler for MapReduce CoGRS attempts to schedule every reduce task, R, at its center-of-gravity node determined by:  The network locations of R’s feeding nodes  The skew in the sizes of R’s partitions By scheduling reduce tasks at their center-of-gravity nodes, we argue for diminished network traffic and improved Hadoop performance 7

8. Talk Roadmap The proposed Center-of-Gravity Reduce Scheduler (CoGRS) Tradeoffs: – Locality, Concurrency, Load Balancing and Utilization CoGRS and the Shuffle Stage in Hadoop Quantitative Methodology and Evaluations CoGRS on a Private Cloud CoGRS on Amazon EC2 Concluding Remarks 8

9. CoGRS Approach To address data locality and partitioning skew, CoGRS attempts to place every reduce task, R, at a suitable node that minimizes: – Total Network Distance of R (TND R ) – Shuffle Data We suggest that a suitable node would be the center-of-gravity node in accordance with: – The network locations of R’s feeding nodes – The weights of R’s partitions We define the weight of a partition P needed by R as the size of P divided by the total sizes of all the partitions needed by R 1 2 To address 1 2 9

10. Weighted Total Network Distance We propose a new metric called Weighted Total Network Distance (WTND) and define it as follows: – WTND R = where: n is the number of R’s partition, ND is the network distance required to shuffle a partition i to R, and w i is the weight of a partition i In principle, the center-of-gravity of R is always one of R’s feeding nodes since it is less expensive to access data locally than to shuffle them over the network Hence, we designate the center-of-gravity of R to be the feeding node of R that provides the minimum WTND R 10

11. Locality, Load Balancing, Concurrency, and Utilization Tradeoffs Strictly exploiting data locality can lead to scheduling skew GoGRS gives up some locality for the sake of extracting more concurrency and improving load balancing and cluster utilization CS RS1 TT1 RS2 TT2 TT3 TT4 TT5 JT R1 R2 R3 R4 IDLE CS RS1 TT1 RS2 TT2 TT3 TT4 TT5 JT R1 R2 R3 R4 Better Utilization and Load Balancing isOccupied? YES  Attempt to Schedule Close to TT5 11

12. GoGRS and Early Shuffle To determine the center-of-gravity node of a particular reduce task, R, we need to designate the network locations of R’s feeding nodes This cannot be precisely determined before the Map phase commits because any map task could lead a cluster node to become a feeding node of R Default Hadoop starts scheduling reduce tasks after only 5% of map tasks commit so as to overlap Map and Reduce phases GoGRS defers early shuffle a little bit (e.g., after 20% of map tasks commit) so that most (or all) keys (which determine reduce tasks) will likely be encountered EARLY SHUFFLE Shuffle Map Reduce 12

13. Quantitative Methodology Benchmark Sort1Sort2WordCountK-Means Key Frequency UniformNon-UniformReal Log FilesRandom Data Distribution UniformNon-UniformReal Log FilesRandom Dataset Size 14GB13.8GB11GB5.2GB Map Tasks 2382281184 Reduce Tasks 25 33 We evaluate CoGRS on: – A private cloud with 14 machines – Amazon EC2 with 8, 16, and 32 instances We use Apache Hadoop 0.20.2 We use various benchmarks with different dataset distributions 13

14. Timeline: Sort2 (An Example) H_OFF Ends H_ON Ends Earlier GoGRS Ends Even Earlier Defers early shuffle a little bit 14

15. Reduce Network Traffic on Private Cloud On average, CoGRS maximizes node-local data by 34.5% and minimizes off-rack data by 9.6% versus native Hadoop 15

16. Execution Times on Private Cloud CoGRS outperforms native Hadoop by an average of 3.2% and by up to 6.3% 16

17. GoGRS on Amazon EC2: Sort2 Compared to native Hadoop, on average, CoGRS maximizes node-local data by 1%, 32%, and 57.9%, and minimizes off-rack data by 2.3%, 10%, and 38.6% with 8, 16, and 32 cluster sizes, respectively 8 EC2 Instances16 EC2 Instances32 EC2 Instances This translates to 1.9%, 7.4%, and 23.8% average reductions in job execution times under CoGRS versus native Hadoop with 8, 16, and 32 cluster sizes, respectively 17

18. Concluding Remarks In this work we observed that the network load is of special concern with MapReduce  Large amount of traffic can be generated during the shuffle stage  This can deteriorate Hadoop performance We realized that scheduling reduce tasks at their center-of-gravity nodes has positive effects on Hadoop’s network traffic and performance  Average reductions of 9.6% and 38.6% of off-rack network traffic have been accomplished on a private cloud and on Amazon EC2, respectively  This provided Hadoop by up to 6.3% and 23.8% performance improvement on a private cloud and on Amazon EC2, respectively We expect GoGRS to play a major role in MapReduce for applications that exhibit high partitioning skew (e.g., scientific applications) 18

19. Thank You! Questions? 19