CSF4 Meta-Scheduler Tutorial 1st PRAGMA Institute Zhaohui Ding or College of Computer Science & Technology Jilin University National Biomedical Computing Resource, University of California, San Diego

2 Agenda Meta-scheduler & CSF4 Introduction CSF4 Architecture CSF4 Functionalities Future Work Demo and Practice

3 What is Meta-Scheduler Resource Allocation & Management Heterogeneous Distributed Dynamic Local Scheduler VS Meta-scheduler

4 Local Scheduler VS Meta-Scheduler Local SchedulerMeta Scheduler Administrative scope Cluster, Single DomainGrid, Multiple Domains, Virtual Organizations Hardware & Software (OS) HomogeneousHeterogeneous OS-independent Data management LAN file system (NFS, FTP, scp) Global file system (Gridftp, Gfarm) SecurityOS user/passwd, NIS, ssh public key Grid Security Infrastructure(GSI) Resource Management Protocol Specified, Private Protocols for different local scheduler Standard, Open, General- Purpose Protocols (GRAM) Scheduling mode CentralizedCentralized / Distributed

5 Meta-Scheduler VS Local Scheduler Local Scheduler LSF (Load Sharing Facility) PBS (Portable Batch System) SGE (Sun Grid Engine) Condor IBM Loadleveler Meta-Scheduler CSF Maui (Silver) Gridway Nimrod-G Condor-G

6 CSF4 What is CSF Meta-Scheduler Full Name: Community Scheduler Framework CSF4 contains a group of grid services host in GT4 CSF4 is a full WSRF compliant meta-scheduler. Open Source project and can be accessed at Developed by Lab. of Distributed Computing and System Architecture, Jilin University, China CSF4 has been added to Globus Toolkit 4 as an Execution Component

7 CSF4 in Globus Toolkit 4

8 A typical deployment

9 What CSF4 Can Do? Basic Functionalities Submit jobs to Grid without Specifying Cluster Monitor and Control Jobs Provide Queuing Service Schedule jobs and resource by custom-built polices CSF4 Portlet (A Web browser based User Interface)

10 What CSF4 Can Do? (cont.) Advanced Functionalities Multiple Domains Resource Information Sharing Multiple scale resource scheduling policies Automatic user credentials delegation Automatic data-staging Extensible scheduling framework Supporting grid parallel jobs (MPI&MPICH-G2)

11 CSF4 – Architecture

12 CSF4 – Architecture User view

13 Local Scheduler And Infrastructure Supported by CSF4 Local Scheduler Supported LSF PBS SGE Condor Infrastructure Supported Globus Toolkit 4 Globus Toolkit 2

14 CSF4 – Functionalities Scheduling Plug-in Framework Designed For Queuing Service Provide A set of policies Customizable Extensible

15 Existent Scheduling Policies FCFS (First Come First Serve) round-robin Default policy Throttle Restrict the number of jobs in a scheduling cycle Array Job Plug-in Design for life science applications (such as AutoDock, BLAST) MPICH-G2 Plug-in (under-developing) The plug-in guarantee the synchronized resource allocation can be successful Data intensive applications plug-in (under-developing)

16 Schedule plug-in & scheduling policies Each policy is implemented inside a scheduling plugin module Each policy is implemented inside a scheduling plugin module A queue can load multiple plugin modules A queue can load multiple plugin modules

17 Resource Information Sharing A MDS information provider for CSF4 Multiple CSF4 can share the resource information

18 CSF4 – Functionalities (cont.) Deploy Multiple CSF4 in a Grid Community Deploy Multiple CSF4 in a Grid Community

19 Array Job AutoDock and Blast-like applications A large number of sub-jobs. Execute same binary Different input/output files

20 Array Job (cont.) Advantages Submit job only once Save submission time and memory storage

21 Data Staging Manual Data Staging Which clusters I can use? Which clusters my jobs will running on? Where is the output data? When will the job finish, so that I can stage-out the output data?

22 Manual Data Staging Without Meta-Scheduler

23 Automatic Data Staging With CSF4 Automatic Data Staging

24 Integrate CSF4 with Gfarm With CSF4 Automatic Data-Staging and Gfarm

25 Application Based Scheduling

26 CSF4 User Interface CSF4 Portal

27 CSF4 User Interface CSF4 Command Line

28 Under-Developing work

29 Demo & Practice

30 Thank you