1. October 19, 2005Charm++ Workshop, 2005 1 Faucets Tutorial Presented by Esteban Pauli and Greg Koenig Parallel Programming Lab, UIUC

2. Charm++ Workshop, 20052 October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

3. Charm++ Workshop, 20053 October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

4. Charm++ Workshop, 20054 October 19, 2005 Current Situation Cluster User Cluster User Where should I submit my job?

5. Charm++ Workshop, 20055 October 19, 2005 Faucets System User Cluster Faucets System I’ll submit my job to Faucets!!!

6. Charm++ Workshop, 20056 October 19, 2005 Faucets System User Faucets System Cluster 1. User submits job to Faucets system Job requirements

7. Charm++ Workshop, 20057 October 19, 2005 Faucets System User Faucets System Cluster 2. Faucets forwards requests to clusters meeting minimum requirements Job requirements

8. Charm++ Workshop, 20058 October 19, 2005 Faucets System User Faucets System Cluster 3. Clusters analyze job specs and return bids Bid Bids

9. Charm++ Workshop, 20059 October 19, 2005 Faucets System User Faucets System Cluster 4. User selects winner Winner selected

10. Charm++ Workshop, 200510 October 19, 2005 Faucets System User Faucets System Cluster 5. Winner runs job, user monitors progress

11. Charm++ Workshop, 200511 October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

12. Charm++ Workshop, 200512 October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

13. Charm++ Workshop, 200513 October 19, 2005 Why a New Scheduler? Current schedulers try to maximize throughput – good for showing cluster is busy, but can be bad for users Users worry about deadlines, priorities, fairness, etc. Profit centers worry about profits Need good interface between cluster scheduler and meta scheduler Want scheduler that can leverage run-time system (Charm++ checkpoint/restart, shrink/expand)

14. Charm++ Workshop, 200514 October 19, 2005 Scheduler Database Request Server Job Monitor Cluster Monitor Strategy Job scheduler Scheduler Design User Cluster

15. Charm++ Workshop, 200515 October 19, 2005 Installation Install MySQL database Install Charm++, MPI./configure; (edit config file); make; make install sh sqlwriter.sh | mysql –user=root –p Set up node list./startScheduler Done!!!

16. Charm++ Workshop, 200516 October 19, 2005 Job Submission ufrun (uni-processor), frun (Charm++), frun-mpi (mpi) used for interactive jobs ufsub, fsub, fsub-mpi used for batch jobs Options: +n, +p, +ppn, -stdout, … See manual for full details

17. Charm++ Workshop, 200517 October 19, 2005 Job Control Job monitoring: fjobs  Options: -user, -full, id  Sample output: UserId JobId # Nodes Head Node Status Name Time Remaining ------ ----- ------- --------- ------ ---- -------------- fuser 63503 1 arch017.c RUNNING my_exe 0:06:39:28 Job deletion: fkill  Options: -u, id

18. Charm++ Workshop, 200518 October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

19. Charm++ Workshop, 200519 October 19, 2005 Disclaimer Current code base developed as a proof of concept Code is not yet production quality  Code works, but has not been tested thoroughly  Code has some security issues Use at your own risk, and please report bugs Code will be updated within the next year

20. Charm++ Workshop, 200520 October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

21. Charm++ Workshop, 200521 October 19, 2005 Central Server Responsible for keeping all information about users and clusters in system Responsible for forwarding users’ job requests to clusters Responsible for dispute arbitration Responsible for keeping account balances

22. Charm++ Workshop, 200522 October 19, 2005 Central Server: Database Cluster table: contains information about confederated clusters mysql> create table Cluster ( domainName text not null, port int not null, status text not null, acctId int not null );

23. Charm++ Workshop, 200523 October 19, 2005 Central Server: Database (cont.) User table: contains information about registered users mysql> create table users ( userid text not null, password text not null, localCluster text not null, acctId int not null );

24. Charm++ Workshop, 200524 October 19, 2005 Central Server: Database (cont.) Account table: keeps account balances for clusters and users mysql> create table accounts ( clusterid text not null, acctId int not null, balance int not null, pbalance int not null);

25. Charm++ Workshop, 200525 October 19, 2005 Central Server: Database (cont.) Job table: keeps track of all running and completed jobs mysql> create table Jobs ( JobID text not null, User text not null, Status text not null, ClusterID text not null);

26. Charm++ Workshop, 200526 October 19, 2005 Central Server: Installation Compilation & Configuration:  cd faucets; make  Edit faucets/cs/db.properties  Get and install JDBC Running the central server:  cd faucets  java -cp.:/path/to/mm.mysql-2.0.8-bin.jar TheServer As users and clusters join, update DB

27. Charm++ Workshop, 200527 October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

28. Charm++ Workshop, 200528 October 19, 2005 Cluster Daemon Purpose: provide interface between central server and cluster scheduler No user intervention Installation: cd faucets; make Usage:  cd faucets  java -cp.:./common/TB.jar cd.ClusterDaemon /tmp/

29. Charm++ Workshop, 200529 October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

30. Charm++ Workshop, 200530 October 19, 2005 Command Client Installation: cd faucets/cc; make Some common commands  Job submission java cc.FaucetCLI [-input file1,file2,...,filen] [ ]  Retrieving output files java FaucetCLI GetFile

31. Charm++ Workshop, 200531 October 19, 2005 Faucets GUI

32. Charm++ Workshop, 200532 October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

33. Charm++ Workshop, 200533 October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

34. Charm++ Workshop, 200534 October 19, 2005 Faucets Scheduling Framework In many ways Faucets can be thought of simply as a framework for creating cluster scheduling solutions Any Faucets deployment has some scheduling objective that it tries to achieve  Traditional FIFO scheduling  On-demand scheduling – driven by workloads and the priorities that users have to access resources  Resource bartering – driven on an economic basis (“where can this job be run for the least cost?”)

35. Charm++ Workshop, 200535 October 19, 2005 Schedule Strategies The scheduling method used by Faucets can readily be changed by writing a scheduling strategy The scheduling strategy can do interesting things that take advantage of features in lower level runtime systems (e.g., Charm++ Adaptive Jobs that shrink/expand to better utilize cluster) Scheduling strategy code is a C++ class  Implement to reflect the scheduling method  Recompile the cluster scheduler executable

36. Charm++ Workshop, 200536 October 19, 2005 Schedule Strategy Examples PriorityFIFOStrategy  Jobs have assigned priorities  Jobs of a given priority are scheduled FIFO  Jobs of a higher priority can preempt jobs of a lower priority LimitFIFOStrategy  Jobs are scheduled FIFO  The number of short/long jobs that a given user may be running simultaneously is limited to prevent resource domination GanttChartStrategy  Jobs are scheduled by arranging them on a Gantt Chart  Predictions can be made about whether a given job can be completed before a user-specified deadline

37. Charm++ Workshop, 200537 October 19, 2005 Implementing a Strategy (1) To implement a new strategy  Inherit from SchedulingStrategy base class  Implement four methods class PriorityFIFOStrategy : public SchedulingStrategy { public: PriorityFIFOStrategy (int nodes); float is_available (Job *job, Job *wait_queue, Job *run_queue); void allocate_nodes (Job *wait_queue, Job *run_queue); void addjob (char *username, int num_procs); void removejob (char *username, int num_procs); };

38. Charm++ Workshop, 200538 October 19, 2005 Implementing a Strategy (2) float is_available (Job *job, Job *wait_queue, Job *run_queue); Method parameters  job – a pointer to an incoming job  wait_queue – a pointer to the queue of all waiting jobs  run_queue – a pointer to the queue of all running jobs Method returns a float  0.0 to 1.0 – the utilization of the cluster if the incoming job is accepted for execution (0% to 100%)  -1.0 - the incoming job cannot be accepted for immediate execution This method is used (indirectly) by the Cluster Daemon to make the choice of target cluster when doing metascheduling

39. Charm++ Workshop, 200539 October 19, 2005 Implementing a Strategy (3) void allocate_nodes (Job *wait_queue, Job *run_queue); Method parameters  wait_queue – a pointer to the queue of all waiting jobs  run_queue – a pointer to the queue of all running jobs Method examines each Job object in the wait_queue and updates it by allocating available cluster nodes to the job so the scheduler can launch it  For Charm++ jobs, normally try to allocate max_nodes first and then min_nodes after that to fulfill the request

40. Charm++ Workshop, 200540 October 19, 2005 Implementing a Strategy (4) void addjob (char *username, int num_procs); void removejob (char *username, int num_procs); Method parameters  username – the user submitting a new job into the cluster  num_procs – the number of processors the new job is allocated Method can be used to enforce some characteristic about the number of processors allocated to any given user (e.g., limit the total number of processors that any given user is allocated)

41. Charm++ Workshop, 200541 October 19, 2005 Instantiating a Strategy The scheduling strategy is instantiated in the Scheduler class constructor located in Scheduler.C strategy = new PriorityFIFOStrategy (num_nodes); The total number of nodes in the cluster is provided to the strategy’s constructor That’s it! The scheduler code calls into your custom Strategy class whenever it needs to make a decision about whether a new job can be scheduled, to allocate nodes to the job, etc.

42. Charm++ Workshop, 200542 October 19, 2005 Questions?

43. Charm++ Workshop, 200543 October 19, 2005 Thanks