1. October 18, 2005 Charm++ Workshop 2005 1 Faucets A Framework for Developing Cluster and Grid Scheduling Solutions Presented by Esteban Pauli Parallel Programming Lab, UIUC

2. October 18, 2005 Charm++ Workshop 2005 2 Outline Motivation and Goals Motivation and Goals System Overview System Overview Meta Scheduler Meta Scheduler Cluster Scheduler Cluster Scheduler Conclusions Conclusions Future Work Future Work

3. October 18, 2005 Charm++ Workshop 2005 3 Motivation Clusters are becoming ubiquitous Clusters are becoming ubiquitous Workloads come in bursts, resulting in alternation between low and high utilization Workloads come in bursts, resulting in alternation between low and high utilization Need framework for sharing computing power Need framework for sharing computing power Traditional schedulers care about throughput, not deadlines, priorities, etc. Traditional schedulers care about throughput, not deadlines, priorities, etc.

4. October 18, 2005 Charm++ Workshop 2005 4 Goals Provide technical and economic framework for allowing organizations to share their resources (clusters) Provide technical and economic framework for allowing organizations to share their resources (clusters) Provide new cluster scheduler which facilitates the above Provide new cluster scheduler which facilitates the above Provide platform for implementing new scheduling strategies Provide platform for implementing new scheduling strategies

5. October 18, 2005 Charm++ Workshop 2005 5 System Overview Faucets consists of two main components: meta scheduler and cluster scheduler Faucets consists of two main components: meta scheduler and cluster scheduler Meta scheduler provides mechanism for discovering and sharing resourcesMeta scheduler provides mechanism for discovering and sharing resources Cluster scheduler makes scheduling decisions based on local and global workloadCluster scheduler makes scheduling decisions based on local and global workload Components interact to meet users’ needs Components interact to meet users’ needs

6. October 18, 2005 Charm++ Workshop 2005 6 Central Server Database Cluster Cluster Daemon Scheduler System Architecture User Cluster Cluster Daemon Scheduler User

7. October 18, 2005 Charm++ Workshop 2005 7 Outline Motivation and Goals Motivation and Goals System Overview System Overview Meta Scheduler Meta Scheduler Cluster Scheduler Cluster Scheduler Conclusions Conclusions Future Work Future Work

8. October 18, 2005 Charm++ Workshop 2005 8 The Faucets Meta Scheduler Job Monitor Job Submission Job Specs Bids Job Specs Job Id Cluster

9. October 18, 2005 Charm++ Workshop 2005 9 The Faucets Meta Scheduler Users provide job requirements Users provide job requirements System requirements: architecture, number of processors, minimum memory, etc.System requirements: architecture, number of processors, minimum memory, etc. Software requirements: utilities, dynamic libraries, packages, etc.Software requirements: utilities, dynamic libraries, packages, etc. Contract requirements: deadline, reliability, maximum price, etc.Contract requirements: deadline, reliability, maximum price, etc. Use XML, easily expandableUse XML, easily expandable Clusters bid on job Clusters bid on job Winning bidder executes job Winning bidder executes job

10. October 18, 2005 Charm++ Workshop 2005 10 The Faucets Meta Scheduler Bidding requires no user intervention Bidding requires no user intervention Clusters bid on jobs based on current conditions Clusters bid on jobs based on current conditions Local utilizationLocal utilization Account balancesAccount balances Depending on scheduling strategy, might not be able to accept all jobs Depending on scheduling strategy, might not be able to accept all jobs

11. October 18, 2005 Charm++ Workshop 2005 11 The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (10000) Bob (100) Cluster 2 Shared (- 200) John (0)Joan (0)

12. October 18, 2005 Charm++ Workshop 2005 12 The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (10000) Bob (100) Cluster 2 Shared (- 200) John (0)Joan (0) Bob runs job worth 1000 units on Cluster 2

13. October 18, 2005 Charm++ Workshop 2005 13 The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (- 200) John (0)Joan (0) Bob’s account drained, remaining 900 units come from shared pool

14. October 18, 2005 Charm++ Workshop 2005 14 The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (- 200) John (0)Joan (0) Cluster 2’s policy: 50% to shared, rest divided equally

15. October 18, 2005 Charm++ Workshop 2005 15 The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (300) John (250) Joan (250) Cluster 2’s shared balance up 500, John & Joan get 250 each

16. October 18, 2005 Charm++ Workshop 2005 16 The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (300) John (250) Joan (250) Global balance remains unchanged

17. October 18, 2005 Charm++ Workshop 2005 17 The Faucets Meta Scheduler Both users and clusters have account balances Both users and clusters have account balances Cluster administrators decide how to share balance among users Cluster administrators decide how to share balance among users Central Server Cluster 1 Shared (9100) Bob (0) Cluster 2 Shared (300) John (250) Joan (250) Have limits to negative balances to prevent freeloading

18. October 18, 2005 Charm++ Workshop 2005 18 Outline Motivation and Goals Motivation and Goals System Overview System Overview Meta Scheduler Meta Scheduler Cluster Scheduler Cluster Scheduler Conclusions Conclusions Future Work Future Work

19. October 18, 2005 Charm++ Workshop 2005 19 Cluster Scheduler Traditional schedulers concerned only with throughput – try to have highest possible utilization Traditional schedulers concerned only with throughput – try to have highest possible utilization Faucets cluster scheduler provides different strategies to allow more efficient bidding Faucets cluster scheduler provides different strategies to allow more efficient bidding Leverage run-time systems Leverage run-time systems Flexible design allows for easy implementation of new strategies Flexible design allows for easy implementation of new strategies

20. October 18, 2005 Deadline-Driven Scheduling (Gantt Chart)   Schedule based on #processors, deadline, wall-time   As new jobs arrive, reschedule meeting all demands  Allows bidding based on deadline – can charge different amounts based on user’s flexibility  Can leverage Charm++ runtime system to shrink and expand jobs Job 1, 4 PE’s, 4 time slices Job 2, 2 PE’s, 6 time slices Job 3, 3 PE’s, 3 time slices New Job, 2 PE’s, 7 time slices P1 P2 P3 P4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Original schedule P1 P2 P3 P4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 New schedule

21. October 18, 2005 Priority-Driven Scheduling Leverage Charm++ and other checkpoint/restart mechanisms Leverage Charm++ and other checkpoint/restart mechanisms Priority can be based on rank (military, institutional, etc), price paid, or other factors Priority can be based on rank (military, institutional, etc), price paid, or other factors P1 P2 P3 P4 Original Schedule Job 1, normal priority Job 2, normal priority Job 3, high priority Job 3 arrives Job 3 terminates P1 P2 P3 P4 New Schedule

22. October 18, 2005 Charm++ Workshop 2005 22 Outline Motivation and Goals Motivation and Goals System Overview System Overview Meta Scheduler Meta Scheduler Cluster Scheduler Cluster Scheduler Conclusions Conclusions Future Work Future Work

23. October 18, 2005 Charm++ Workshop 2005 23 Conclusions Clusters becoming more common, Faucets provides economic and technical framework for sharing Clusters becoming more common, Faucets provides economic and technical framework for sharing Flexible cluster scheduler allows scheduling based on deadlines, priorities, etc. Flexible cluster scheduler allows scheduling based on deadlines, priorities, etc. Cluster scheduler leverages run-time systems to increase functionality Cluster scheduler leverages run-time systems to increase functionality

24. October 18, 2005 Charm++ Workshop 2005 24 Future Work NCSA Faculty Fellowship – on- demand access NCSA Faculty Fellowship – on- demand access How do we control anonymous access? How do we control anonymous access? Only allow pre-selected applicationsOnly allow pre-selected applications Virtual machines (Xen, VMWare, etc.)Virtual machines (Xen, VMWare, etc.) Leverage virtualization to allow processor sharing Leverage virtualization to allow processor sharing Re-architect Faucets to make more robust, easier to write strategies Re-architect Faucets to make more robust, easier to write strategies

25. October 18, 2005 Charm++ Workshop 2005 25 Questions?

26. October 18, 2005 Charm++ Workshop 2005 26 Thanks!