Reining in the Outliers in Map-Reduce Clusters using Mantri Ganesh Ananthanarayanan, Srikanth Kandula, Albert Greenberg, Ion Stoica, Yi Lu, Bikas Saha, Edward Presenter: Weiyue Xu OSDI'10 Proceedings of the 9th USENIX conference on Operating systems design and implementation

Credits Modified version of – ures/ ures/ –research.microsoft.com/en- us/UM/people/srikanth/data/Combating%20Outliers %20in%20Map-Reduce.web.pptxresearch.microsoft.com/en- us/UM/people/srikanth/data/Combating%20Outliers %20in%20Map-Reduce.web.pptx

Outline Introduction Cause of the Outlier Mantri Evaluation Discussion and Conclusion

log(size of dataset) GB 10 9 TB PB EB log(size of cluster) HPC, || databases mapreduce MapReduce Decouples customized data operations from mechanisms to scale Widely used Cosmos (based on SVC ’ s Dryad) + Bing Google Hadoop inside Yahoo! and on Amazon ’ s Cloud (AWS) e.g., the Internet, click logs, bio/genomic data

Local write An Example How it Works: Goal Find frequent search queries to Bing SELECT Query, COUNT(*) AS Freq FROM QueryTable HAVING Freq > X What the user says: ReadMap Reduce file block 0 job manager task output block 0 output block 1 file block 1 file block 2 file block 3 assign work, get progress

Outliers slow down map-reduce jobs Map.Read 22K Map.Move 15K Map 13K Reduce 51K Barrier File System We find that: 6 Tackling outliers, we can speed up jobs while using resources efficiently: Quicker response improves productivity Predictability supports SLAs Better resource utilization

From a phase to a job A job may have many phases An outlier in an early phase has a cumulative effect Data loss may cause multi-phase recompute  outliers

Delay due to a recompute readily cascades Why outliers? reduce sort Delay due to a recompute map 8 Due to unavailable input, tasks have to be recomputed

Outlier stragglers : Tasks that take  1.5 times the median task in that phase recomputes : Tasks that are re-run because their output was lost 50% phases have > 10% stragglers and no recomputes 10% of the stragglers take >10X longer Frequency of Outliers straggler

At median, jobs slowed down by 35% due to outliers Cost of Outliers

Previous solution The original MapReduce paper observed the problem and did not solve it in depth Current schemes (e.g. Hadoop, LATE) duplicate long- running tasks based on some metrics Drawbacks –Some may be unnecessary –Use extra resources –Placement may be a problem

What this Paper is About Identify fundamental causes of outliers Mantri: A Cause-, Resource-Aware mitigation scheme –Case by case analysis: takes distinct actions based on cause –Considers opportunity cost of actions Results from a production deployment

Causes of Outliers Data Skew: data size varies across tasks in a phase.

Reduce task Map output uneven placement is typical in production reduce tasks are placed at first available slots Crossrack Traffic Causes of Outliers Rack

50% of recomputes happen on 5% of the machines Recompute increases resource usage Bad and Busy Machines Causes of Outliers

70% of cross track traffic is reduce traffic Reduce reads from every map, tasks in a spot with slow network run slower Tasks compete network among themselves 50% phases takes 62% longer to finish than ideal placement Causes of Outliers

Outliers cluster by time –Resource contention might be the cause Recomputes cluster by machines –Data loss may cause multiple recomputes

Mantri Cause aware, and resource aware Fix each problem with different strategies Runtime = f (input, network, dataToProcess,...)

Delay due to a recompute readily cascades Why outliers? reduce sort Delay due to a recompute map 19 Mantri [Avoid Recomputations]

Idea: Replicate intermediate data, use copy if original is unavailable Challenge: What data to replicate? Insight: Cost to recompute vs cost to replicate M1M1 M2M2 t redo = r 2 (t 2 +t 1 redo ) Cost to recompute depends on data loss probabilities and time taken, and also recursively looks at prior phases. t redo > t rep Mantri preferentially acts on more costly inputs 20 t = predicted runtime of task r = predicted probability of recomputation at machine

Reduce task Map output uneven placement is typical in production reduce tasks are placed at first available slots Crossrack Traffic Rack Mantri [Network Aware Placement]

Idea: Avoid hot-spots, keep traffic on a link proportional to bandwidth Challenges: Global co-ordination, congestion detection Insights: –Local control is a good approximation(each job balances its traffic) –Link utilizations average out on the long task and are steady on the short task If rack i has d i map output need d u i and d v i for upload and download while b u i and b v i bandwidths available Place a i fraction of reduces such that: 22

Data Skew - About 25% of outliers occur due to more dataToProcess (workload imbalance) Ignoring these is better than duplicating (state-of-the-art) Mantri [Data Aware Task Ordering]

Problem: Workload imbalance causes tasks to straggle. Idea: Restarting outliers that are lengthy is counter- productive. Insights: –Theorem [Graham, 1969] –Scheduling tasks with longest processing time first is at-most 33% worse than optimal schedule. 24 Builds an estimator T ~ function (dataToProcess) We schedule tasks in descending order of dataToProcess Theorem [due to Graham, 1969] Doing so is no more than 33% worse than the optimal

Mantri [Resource Aware Restart] Problem: 25% outliers remain, likely due to Idea: Restart tasks elsewhere in the cluster asap Challenge: restart or duplicate? 25 (a) (b) (c) Running task Potential restart (t new ) now time t rem √ × × Save time and resources iff P(c t rem > (c+1) t new ) P(c t rem > (c+1) t new ) > δ If pending work, duplicate iff save both time and resource Else, duplicate if expected savings are high Continuously observe and kill wasteful copies

26 Summary Reduce recomputation: –preferentially replicate costly-to-recompute tasks Poor network: –each job locally avoids network hot-spots DataToProcess: –schedule in descending order of data size Bad machines: – quarantine persistently faulty machines Others: –restart or duplicate tasks, cognizant of resource cost.

Evaluation Methodology Mantri run on production clusters Baseline is results from Dryad Use trace-driven simulations to compare with other systems

Comparing Jobs in the Wild w/ and w/o Mantri for one month of jobs in Bing production cluster 340 jobs that each repeated at least five times during May (release) vs. Apr 1-30 (pre-release)

In Production, Restarts …

In Trace-Replay Simulations, Restarts … CDF % cluster resources

Protecting Against Recomputes CDF % cluster resources

Conclusion Outliers are a significant problem Happens due to many causes Mantri: cause and resource aware mitigation outperforms prior schemes 32

Discussion Mantri does case by case analysis for each cause, what if the causes are inter-dependent?

Questions or Comments? Thanks!

Estimation of t rem and t new d: input data size d read : the amount read

Estimation of t new processRate: estimated of all tasks in the phase locationFactor: machine performance d: input size