October 19, 2005Charm++ Workshop, Faucets Tutorial Presented by Esteban Pauli and Greg Koenig Parallel Programming Lab, UIUC

Charm++ Workshop, October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

Charm++ Workshop, October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

Charm++ Workshop, October 19, 2005 Current Situation Cluster User Cluster User Where should I submit my job?

Charm++ Workshop, October 19, 2005 Faucets System User Cluster Faucets System I’ll submit my job to Faucets!!!

Charm++ Workshop, October 19, 2005 Faucets System User Faucets System Cluster 1. User submits job to Faucets system Job requirements

Charm++ Workshop, October 19, 2005 Faucets System User Faucets System Cluster 2. Faucets forwards requests to clusters meeting minimum requirements Job requirements

Charm++ Workshop, October 19, 2005 Faucets System User Faucets System Cluster 3. Clusters analyze job specs and return bids Bid Bids

Charm++ Workshop, October 19, 2005 Faucets System User Faucets System Cluster 4. User selects winner Winner selected

Charm++ Workshop, October 19, 2005 Faucets System User Faucets System Cluster 5. Winner runs job, user monitors progress

Charm++ Workshop, October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

Charm++ Workshop, October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

Charm++ Workshop, October 19, 2005 Why a New Scheduler? Current schedulers try to maximize throughput – good for showing cluster is busy, but can be bad for users Users worry about deadlines, priorities, fairness, etc. Profit centers worry about profits Need good interface between cluster scheduler and meta scheduler Want scheduler that can leverage run-time system (Charm++ checkpoint/restart, shrink/expand)

Charm++ Workshop, October 19, 2005 Scheduler Database Request Server Job Monitor Cluster Monitor Strategy Job scheduler Scheduler Design User Cluster

Charm++ Workshop, October 19, 2005 Installation Install MySQL database Install Charm++, MPI./configure; (edit config file); make; make install sh sqlwriter.sh | mysql –user=root –p Set up node list./startScheduler Done!!!

Charm++ Workshop, October 19, 2005 Job Submission ufrun (uni-processor), frun (Charm++), frun-mpi (mpi) used for interactive jobs ufsub, fsub, fsub-mpi used for batch jobs Options: +n, +p, +ppn, -stdout, … See manual for full details

Charm++ Workshop, October 19, 2005 Job Control Job monitoring: fjobs  Options: -user, -full, id  Sample output: UserId JobId # Nodes Head Node Status Name Time Remaining fuser arch017.c RUNNING my_exe 0:06:39:28 Job deletion: fkill  Options: -u, id

Charm++ Workshop, October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

Charm++ Workshop, October 19, 2005 Disclaimer Current code base developed as a proof of concept Code is not yet production quality  Code works, but has not been tested thoroughly  Code has some security issues Use at your own risk, and please report bugs Code will be updated within the next year

Charm++ Workshop, October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

Charm++ Workshop, October 19, 2005 Central Server Responsible for keeping all information about users and clusters in system Responsible for forwarding users’ job requests to clusters Responsible for dispute arbitration Responsible for keeping account balances

Charm++ Workshop, October 19, 2005 Central Server: Database Cluster table: contains information about confederated clusters mysql> create table Cluster ( domainName text not null, port int not null, status text not null, acctId int not null );

Charm++ Workshop, October 19, 2005 Central Server: Database (cont.) User table: contains information about registered users mysql> create table users ( userid text not null, password text not null, localCluster text not null, acctId int not null );

Charm++ Workshop, October 19, 2005 Central Server: Database (cont.) Account table: keeps account balances for clusters and users mysql> create table accounts ( clusterid text not null, acctId int not null, balance int not null, pbalance int not null);

Charm++ Workshop, October 19, 2005 Central Server: Database (cont.) Job table: keeps track of all running and completed jobs mysql> create table Jobs ( JobID text not null, User text not null, Status text not null, ClusterID text not null);

Charm++ Workshop, October 19, 2005 Central Server: Installation Compilation & Configuration:  cd faucets; make  Edit faucets/cs/db.properties  Get and install JDBC Running the central server:  cd faucets  java -cp.:/path/to/mm.mysql bin.jar TheServer As users and clusters join, update DB

Charm++ Workshop, October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

Charm++ Workshop, October 19, 2005 Cluster Daemon Purpose: provide interface between central server and cluster scheduler No user intervention Installation: cd faucets; make Usage:  cd faucets  java -cp.:./common/TB.jar cd.ClusterDaemon /tmp/

Charm++ Workshop, October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

Charm++ Workshop, October 19, 2005 Command Client Installation: cd faucets/cc; make Some common commands  Job submission java cc.FaucetCLI [-input file1,file2,...,filen] [ ]  Retrieving output files java FaucetCLI GetFile

Charm++ Workshop, October 19, 2005 Faucets GUI

Charm++ Workshop, October 19, 2005 Cluster Cluster Daemon Scheduler System Architecture Central Server Database User

Charm++ Workshop, October 19, 2005 Outline System Overview Cluster Scheduler Meta Scheduler Writing a Scheduling Strategy

Charm++ Workshop, October 19, 2005 Faucets Scheduling Framework In many ways Faucets can be thought of simply as a framework for creating cluster scheduling solutions Any Faucets deployment has some scheduling objective that it tries to achieve  Traditional FIFO scheduling  On-demand scheduling – driven by workloads and the priorities that users have to access resources  Resource bartering – driven on an economic basis (“where can this job be run for the least cost?”)

Charm++ Workshop, October 19, 2005 Schedule Strategies The scheduling method used by Faucets can readily be changed by writing a scheduling strategy The scheduling strategy can do interesting things that take advantage of features in lower level runtime systems (e.g., Charm++ Adaptive Jobs that shrink/expand to better utilize cluster) Scheduling strategy code is a C++ class  Implement to reflect the scheduling method  Recompile the cluster scheduler executable

Charm++ Workshop, October 19, 2005 Schedule Strategy Examples PriorityFIFOStrategy  Jobs have assigned priorities  Jobs of a given priority are scheduled FIFO  Jobs of a higher priority can preempt jobs of a lower priority LimitFIFOStrategy  Jobs are scheduled FIFO  The number of short/long jobs that a given user may be running simultaneously is limited to prevent resource domination GanttChartStrategy  Jobs are scheduled by arranging them on a Gantt Chart  Predictions can be made about whether a given job can be completed before a user-specified deadline

Charm++ Workshop, October 19, 2005 Implementing a Strategy (1) To implement a new strategy  Inherit from SchedulingStrategy base class  Implement four methods class PriorityFIFOStrategy : public SchedulingStrategy { public: PriorityFIFOStrategy (int nodes); float is_available (Job *job, Job *wait_queue, Job *run_queue); void allocate_nodes (Job *wait_queue, Job *run_queue); void addjob (char *username, int num_procs); void removejob (char *username, int num_procs); };

Charm++ Workshop, October 19, 2005 Implementing a Strategy (2) float is_available (Job *job, Job *wait_queue, Job *run_queue); Method parameters  job – a pointer to an incoming job  wait_queue – a pointer to the queue of all waiting jobs  run_queue – a pointer to the queue of all running jobs Method returns a float  0.0 to 1.0 – the utilization of the cluster if the incoming job is accepted for execution (0% to 100%)  the incoming job cannot be accepted for immediate execution This method is used (indirectly) by the Cluster Daemon to make the choice of target cluster when doing metascheduling

Charm++ Workshop, October 19, 2005 Implementing a Strategy (3) void allocate_nodes (Job *wait_queue, Job *run_queue); Method parameters  wait_queue – a pointer to the queue of all waiting jobs  run_queue – a pointer to the queue of all running jobs Method examines each Job object in the wait_queue and updates it by allocating available cluster nodes to the job so the scheduler can launch it  For Charm++ jobs, normally try to allocate max_nodes first and then min_nodes after that to fulfill the request

Charm++ Workshop, October 19, 2005 Implementing a Strategy (4) void addjob (char *username, int num_procs); void removejob (char *username, int num_procs); Method parameters  username – the user submitting a new job into the cluster  num_procs – the number of processors the new job is allocated Method can be used to enforce some characteristic about the number of processors allocated to any given user (e.g., limit the total number of processors that any given user is allocated)

Charm++ Workshop, October 19, 2005 Instantiating a Strategy The scheduling strategy is instantiated in the Scheduler class constructor located in Scheduler.C strategy = new PriorityFIFOStrategy (num_nodes); The total number of nodes in the cluster is provided to the strategy’s constructor That’s it! The scheduler code calls into your custom Strategy class whenever it needs to make a decision about whether a new job can be scheduled, to allocate nodes to the job, etc.

Charm++ Workshop, October 19, 2005 Questions?

Charm++ Workshop, October 19, 2005 Thanks