4. Why static is bad! Hadoop Pregel MPI Shared cluster Today: static partitioningWant dynamic sharing

6. Comparing Sharing Frameworks: choice Choice of resources Can a framework pick between all resources? A predefined subset? Or a random chosen subset? Why important? Policies may need to be global --localization If you can preempt you can get your preference

7. Comparing Sharing Frameworks: Interference Can frameworks tray to use the same machines? Can a framework pick between all resources? How to avoid this? Offer resources to machines one at a time Statically partition Offer in parallel and arbitrate when conflict arises.

8. Comparing Sharing Frameworks: Granularity Allocation Granularity MPI tasks: gang-schedule, job can’t run until all slots are acquired. Hadoop: elastic, job can start running when it allocates a few slots Why important? If gang-scheduling, then the framework will hoard until it gets all the slots it needs. The cluster may or may not be underutilized. Cluster-wide behaviors

9. Mesos

10. Other Benefits of Mesos Run multiple instances of the same framework »Isolate production and experimental jobs »Run multiple versions of the framework concurrently Build specialized frameworks targeting particular problem domains »Better performance than general-purpose abstractions

11. Goals High utilization of resources Support diverse frameworks (current & future) Scalability to 10,000’s of nodes Reliability in face of failures Resulting design: Small microkernel-like core that pushes scheduling logic to frameworks

12. Design Elements Fine-grained sharing: »Allocation at the level of tasks within a job »Improves utilization, latency, and data locality Resource offers: »Simple, scalable application-controlled scheduling mechanism

13. Element 1: Fine-Grained Sharing Framework 1 Framework 2 Framework 3 Coarse-Grained Sharing (HPC):Fine-Grained Sharing (Mesos): + Improved utilization, responsiveness, data locality Storage System (e.g. HDFS) Fw. 1 Fw. 3 Fw. 2 Fw. 1 Fw. 3 Fw. 2 Fw. 3 Fw. 1 Fw. 2 Fw. 1 Fw. 3 Fw. 2

14. Element 2: Resource Offers Option: Global scheduler »Frameworks express needs in a specification language, global scheduler matches them to resources + Can make optimal decisions – Complex: language must support all framework needs – Difficult to scale and to make robust – Future frameworks may have unanticipated needs

15. Element 2: Resource Offers Mesos: Resource offers »Offer available resources to frameworks, let them pick which resources to use and which tasks to launch + Keeps Mesos simple, lets it support future frameworks - Decentralized decisions might not be optimal

16. Mesos Architecture MPI job MPI scheduler Hadoop job Hadoop scheduler Allocation module Mesos master Mesos slave MPI executor Mesos slave MPI executor task Resource offer Pick framework to offer resources to

17. Mesos Architecture MPI job MPI scheduler Hadoop job Hadoop scheduler Allocation module Mesos master Mesos slave MPI executor Mesos slave MPI executor task Pick framework to offer resources to Resource offer Resource offer = list of (node, availableResources) E.g. { (node1, ), (node2, ) } Resource offer = list of (node, availableResources) E.g. { (node1, ), (node2, ) }

18. Mesos Architecture MPI job MPI scheduler Hadoop job Hadoop scheduler Allocation module Mesos master Mesos slave MPI executor Hadoop executor Mesos slave MPI executor task Pick framework to offer resources to task Framework-specific scheduling Resource offer Launches and isolates executors

19. Drawbacks Poor fairness Jobs with long tasks can dominate There is NO preemption!! Sticky slots Jobs with higher priority can dominate a set of preferred slots Mesos uses lottery scheduling, probability of being offered a slot is proportional to the frameworks priority Head of line blocking Mesos offers resources one framework at a time Prevents frameworks from trying to use the same slots Based on assumptions: scheduling decisions are quick, Mesos revokes offers if a schedules takes too long Essentially leads to a queue

20. Omega

25. Scales Central layer only does optimistic conflict resolution No head of Line blocking Allows for flexible and evolvable scheduling Framework can implement any arbitrary form of scheduling Each framework has global view Frameworks can preempt each other

26. Comparing Sharing Frameworks Choice of resources Interference Allocation Granularity Cluster-wide behaviors

27. Comparing Frameworks