1. Quincy: Fair Scheduling for Distributed Computing Clusters Microsoft Research Silicon Valley SOSP’09 Presented at the Big Data Reading Group by Babu Pillai

2. Motivation Big clusters used for jobs of varying sizes, durations Data from production search cluster @ MS Google: 395s avg. time for Map Reduce job

3. Goal Fine grained sharing of resources, not semi- static allocations Comply with locality constraints – move computation to the data Ensure a notion of fairness – large job should not monopolize cluster at expense of many small jobs

4. Job Model Dryad jobs – root process + set of worker processes, related by a DAG loosely coupled, independently executing processes Model is compatible with Map-Reduce, Hadoop, Condor Model will not work for MPI Will NOT work for SLIPstream jobs May work for Dicer??

5. Locality Cluster modeled as 2 level hierarchy: racks and machines Each worker can have a rack or machine preference based on fraction of data on rack/machine Preference is late bound – when process ready to run

6. Fairness If a job takes t time, and J jobs are running, then it should take no more than Jt Admission control to limit to K concurrent jobs  choice trades off fairness w/locality against avoiding idle resources Queue others, admit in FIFO order as jobs complete Fine grained sharing of whole cluster Assumes only one process per machine

7. Baseline: Queue-based scheduling Queues for each machine, rack, and system-wide Greedy: dispatch from most local queue to keep all machines occupied Greedy Fair: “block” jobs with lots of resources, give priority to unblocked ones Greedy Fair Preemptive: shoot down processes of jobs with more than their fair share, shortest-lived first Hysteresis can be added to GF, GFP to avoid sticky slot issue

8. Big Idea: Graph-based scheduling Represent cluster and process set as a graph Add weights to represent preferences, costs Use solver to convert this to a set of scheduling assignments that satisfy global criteria

9. Quincy Fair Scheduling Encode cluster structure, job tasks as flow graph Edge costs encode policy Cost of tasks to particular machines, racks based on data transfer costs Cost of tasks into U based on time cost of not running Cost out of U can control relative fairness among jobs Time varying costs: when already scheduled, costs of other tasks coming into a compute node go up; cost to U goes up over time Parameterize with 3 variables: cost of waiting in queue, cost of data transfer over rack, core switch

10. Results Unrestricted network case

11. Results Restricted network – much lower bw through core switch

12. Issues / Discussion Assignment of tasks independent – this is not possible for MPI, many VM placement Single dimensional capacities, not easily extended to multidimensional Can support multiple tasks/computer Can adapt K to improve cluster utilization Cannot express constraints such as t1 and t2 should execute on the same rack