1. Universität Dortmund Robotics Research Institute Information Technology Section 03.06.2015 Grid Metaschedulers An Overview and Up-to-date Solutions Christian Grimme christian.grimme@udo.edu

2. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 2 Outline n Grid Metaschedulers and Grid Brokers è A unified view on Grid Scheduling Architecture è Common functions of Grid Scheduling n Up-to-date Grid Scheduling Solutions è CSF (Community Scheduler Framework) è GridWay è Grid Service Broker (in GridBus) è GRMS (GridLab Resource Management System) è Others (Moab/Maui, WMS, Nimrod/G, Condor-G)

3. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 3 Grid Metaschedulers and Grid Brokers n General Problem: Allocation and scheduling of applications in a heterogeneous and dynamically changing set of resources n Three Scenarios: è Enterprise Grid l resources owned by single provider l central broker which interacts with local resource managers è HPC Grid l cooperation of HPC sites to maximize computational power l distributed administrative control, hierarchical brokering è Global Grid l all kind of resources l a broker accepts all kinds of jobs to forward them to other brokers or local resource schedulers

4. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 4 Common Functions of Grid Scheduling n Information è Discover and match resources for applications è Monitor internal strategies, feedback to user n Job Management è Submission of jobs and workflows è Scheduling functionality, forecasting, co-allocation è Reservation management n Security è Rights delegation è Access control n Agreement and Accounting è Service Level Agreements è Data access management

5. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 5 CSF (Community Scheduler Framework) n Open source project as part of the Globus project in cooperation with Platform Computing n Supports implementation of Grid metaschedulers based on GT services (MDS, GRAM, RFT, etc.) n Properties: è Enables communication between heterogeneous schedulers on local level è Allows reservations on resources è Offers simple scheduling mechanisms è Extensible via scheduling plug-ins

6. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 6 CSF Architecture Queuing Service Reservation Service Job Service MetaScheduler GRAM PBSGRAM SGERM Adapter PBSSGELSF Info Service Info Service User Request Scheduling Plugin Specified in RSL Loads queue parameters and associated schedulers Create and administer compute jobs Reserve a resource if supported by LRM Binding to LRM

7. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 7 GridWay n Open source component (Globus Proto Project) n Lightweight metascheduler on top of GT 2.4 – 4.x n Properties: è Support of GGF DRMA standard API for job submission and management è Simple scheduling mechanisms but extensible è Allows job dependencies (workflow) è Supports job migration/adaptive execution (Grid- and application- initiated)

8. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 8 GridWay Architecture Request Manager Dispatch Manager Information Manager Execution Manager Transfer Manager Scheduler GridWay Core Job pool Host pool GRAMRFTMDS Resource GRAMRFTMDS DRMAA LibraryCLI Job control operations Matchmaking, execution and migration Execution of jobs on LRM

9. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 9 GSB – Grid Service Broker (GridBus) n Metascheduler to access computational and data Grids n On top of various middlewares (GT 2.4 – 4.x, Alchemi, Unicore) n Properties: è Collaborates with Grid Economy, Trading and Accounting Services è Interacts with Workflow Management Service è Leverages economic scheduling model

10. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 10 GSB Architecture Parameter Resolver Resource Discovery Grid Scheduler Bookkeeper Actuator & Monitor Data Service Catalogue Resource Catalogue Grid Info Service Agent Local Data Remote Data Host Application Data Parameters Tasks Grid Economy Services Grid Economy Services Workflow Manage- ment Workflow Manage- ment Simulatio n Analysis of job parameters, matchmaking Mapping of jobs to resources Middleware binding for local job execution Wraps and controls job actions (even transfer of distant data)

11. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 11 GRMS - GridLab Resource Management System n Open Source Metascheduler as part of the GridLab project n Based on GT 2.4 (assured to work on GT 4.x) n Properties: è Job dependencies are planned to be supported è Defines a Job Definition Language (GJD) è Simple scheduling mechanisms, extensible è Allows job migration è Provides a GSI enabled Web Service interface

12. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 12 GRMS Architecture Workflow Module Job Queue Broker Module Job Registry Job Manager Resource Discovery Middleware Service Layer GridFTP GRAMMonitoring Other GridLab Services Other GridLab Services Replica Management File Movement... GSI enabled Web Service Interface Matchmaking & Scheduler Plug-ins Jobs in GJD (with dependencies) Queuing of jobs Scheduling and matchmaking

13. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 13 Others n Moab/Maui: è commercial Metascheduler with advanced scheduling an monitoring capabilities è on top of most major scheduling systems n Nimrod/G: è Grid aware solution of Nimrod for compute intensive tasks è works with GT, Legion, Condor n WMS: è no metascheduler, submits only to gLite CEs è has some interesting architecture features (push/pull models) n Condor-G: è simple queue submission system è treats a Grid as local resource

14. AstroGrid-D Meeting 15.11.2006 Grid Metascheduler 14 More information... n Literature: è Tonellotto, N., Yahyapour, R., Wieder, Ph., A Proposal for a Generic Grid Scheduling Architecture, CoreGRID Technical Report 0015, Institute on Resource Management, 2006 è Buyya, R., Venugopal, S., A Gentle Introduction to Grid Computing and Technologies, CSI Communications, 2005 n Web Links: è CSF: www.globus.org/toolkit/docs/4.0/contributions/csf/www.globus.org/toolkit/docs/4.0/contributions/csf/ è GridWay: www.gridway.orgwww.gridway.org è GSB: www.gridbus.orgwww.gridbus.org è GRMS: www.gridlab.orgwww.gridlab.org è Moab/Maui: www.clusterresources.com/pages/products/moab-grid-suite.phpwww.clusterresources.com/pages/products/moab-grid-suite.php è WMS: http://egee-jra1-wm.mi.infn.it/egee-jra1-wm/wms.shtmlhttp://egee-jra1-wm.mi.infn.it/egee-jra1-wm/wms.shtml è Nimrod/G: www.csse.monash.edu.au/~davida/nimrod/index.htmwww.csse.monash.edu.au/~davida/nimrod/index.htm è Condor-G: www.cs.wisc.edu/condor/www.cs.wisc.edu/condor/