EGEE is a project funded by the European Union under contract IST WS-Based Advance Reservation and Co-allocation Architecture Proposal T.Ferrari, E.Ronchieri JRA1 JRA1 all-hands meeting, June

JRA4 meeting, June Contents Use Cases DataGrid WP1 legacy architecture GGF (Grid Resource Allocation and Agreement Protocol) WS-Agreement Specification Web services-based proposal

JRA4 meeting, June Use Cases NETWORK: data replication.  Data replication: guaranteed, dedicated bandwidth to optimize the network performance of a data transfer session (that otherwise would compete with other streams and would get and amount of bandwidth that could greatly vary over time) to support file transfer with deadline (to synchronize job execution with input file transfer) COMPUTING: to reserve resources computing resources (eg. worker nodes) in presence of a large number of other competing jobs STORAGE: to be guaranteed that after the computation phase, a sufficient amount of space is present in a “close SE” to save the output data

JRA4 meeting, June The EDG legacy architecture

JRA4 meeting, June The EDG WP1 legacy architecture 4 1 Creation of reservation Submits reservation request

JRA4 meeting, June The EDG WP1 legacy architecture 5 1 Creation of reservation Starts the discovery phase contacting RB, which returns an ordered list of suitable resources

JRA4 meeting, June The EDG WP1 legacy architecture 6 1 Creation of reservation RA iterates through the list representing suitable resources, and contacts the correspondent RM, until it succeeds

JRA4 meeting, June GGF GRAAP (Grid Resource Allocation and Agreement Protocol) WS-Agreement Conceputal Layered Service Model Reservation Agent Resource Manager

JRA4 meeting, June GGF GRAAP (Grid Resource Allocation and Agreement Protocol) WS-Agreement defines a language and a protocol for Advertising the capabilities of providers Creating agreements based on creational offers Monitoring agreement compliance Agreement Layer (work in progress): Provides a web service based interface to be used to represent and monitor agreements with respect to provisioning of services implemented in the service layer Service Layer (out of the GRAAP scope): Is an application-specific layer of a provided service The interface to this layer is domain-specific May or may not be exposed as a web service interface 1

JRA4 meeting, June GRAAP Agreement structure Agreement Terms Service Description Terms Guarantee Terms Context Name Name: optional identificator Context: participants’ names, lifetime, links to other agreements related to this (co-allocation) Terms: Service Description Terms: - provides information needed to instantiate or identify a service to which this agreement pertains - describes the functionality that will be delivered under an agreement Guaranteed Terms: specify the service level that the parties are agreeing to Terms have to extended for specific usage domains.

JRA4 meeting, June Agreement status SATISFIED VIOLATED at least one term not respected by service provider INACTIVE terms not guaranted ACTIVE mechanism to guarantee the terms in place and running OBSERVED all terms agreed CONSIDERED at least one term under negotiation

JRA4 meeting, June WS-based proposal UI Workload Manager user Co-allocation Agreement Provider Network Agreement Provider Computing Agreement Provider Storage Agreement Provider MM resource ID list Network Resource ID list CE ID listSE ID list CE Service Provider Logging & Bookkeeping NE Service Provider 1 Web service monitor SE Service Provider NE Acceptance SE Acceptance CE Acceptance

JRA4 meeting, June Co-allocation agreement provider Co-allocation Agreement Provider CE ID list Logging & Bookkeeping 1 Co-allocation Agreement Provider - (single reservation) passes the resource ID list to the specific agreement provider - Supports logic for management of co- allocation – Provisioning: in case of concurent allocations – Status: in case of failure of one or more reservations - Provides status information about co- allocations - Returns the co-allocation handle Computing Agreement Provider CE Service Provider monitor CE Acceptance

JRA4 meeting, June Agreement provider Co-allocation Agreement Provider CE ID list Logging & Bookkeeping 1 Agreement Provider - for a list of Resource IDs, it contacts the corresponding service provider and verifies the actual possibility to reserve the service via CA/NE/SE Acceptance - identifies the agreements through a handle - provides information about reservation status - supports protocols to manage the case of a service that is the composition of services independently administered (such as in the case of a network path crossing multiple network administrative domains) - translates the high-level agreement terms specified by the user to a quantitative expression that is understood by the Service Providers Computing Agreement Provider CE Service Provider monitor CE Acceptance

JRA4 meeting, June Acceptance and Service provider Co-allocation Agreement Provider CE ID list Network Agreement Provider NE Service Provider_3 monitor NE Acceptance_2 NE Service Provider_2 monitor NE Acceptance_1 NE Service Provider_n monitor NE Acceptance_n NE Service Provider_1 monitor Acceptance - controls the access to a given resource instance - authentication and authorization - checks the agreement context (eg. the type of service requested to address the right Service Provider if multiple options exist) Service Provider - More than one Service Provider per resource instance possible (for some type of resource such as the network) - It determines if an agreement request can be satisfied (by checking the a slot table DB) -If so, it returns an agreement handle - the monitor provides information about the status of a given active agreement

JRA4 meeting, June Monitoring Two types of Status information needed: - the agreement status -> provided by the Agreement provider - the amount of reserved resources actually used at a given time -> provided by MON Examples of consumers of monitoring information: - the end-user: information directly from the LB - different solution: direct query of the Agreement - Jobs: they need to be informed when an Agreement status changes from INACTIVE to ACTIVE. In this case, a daemon should run on the WN to periodically check the status of a given Agreement. CE Acceptance CE Service Provider MON JC Logging & Bookkeeping LRMS

JRA4 meeting, June Matchmaking The EDG library has to be extended in order to: for each job submission, make use of existing related reservation handles support reservation and co-allocation resource discovery optimize the resource discovery phase with specific policies in case of co-allocation: Examples: – CE and network reservation, outputSE known » Find CEs close to outputSE » For each couple (outputSE, CE_i), find network path – CE, SE and network reservation » Find SEs that support reservation » For each SE_i, find suitable CEs that support reservation » For each couple (SE_i, CE_j), find network path

JRA4 meeting, June Use case 1: job with reserved CE User needs to specify the agreement identifier (ID) associated to the submitted job Once the job is passed to WM, before proceeding with the execution the job, WM needs to verify the status of the Agreement by querying the the LB: WM gets Agreement ID status from LB ; If (not OBSERVED) WM returns an error; else * PUSH mode If (ACTIVE and SATISFIED) WM gets the CE ID associated to the agreement ID from the LB; WM submits the job on CE ID; else hold job in task queue until Agreement status = ACTIVE; * PULL mode WM puts job in TQ; when Agreement status = ACTIVE CE ID gets job associated to relevant agreement ID from TQ NOTE: condor_glidein could be used at the CE Service Provider Level

JRA4 meeting, June Use case 2: Job with reserved SE Submission of job with reserved SE:  USER specifies agreement ID in the JDL  WM queries LB to determine the corresponding SE_ID  MM selects the CE Ids close to SE_ID  Case 1: user wants to replicate some output files automatically JDL contains OutputData A deamon in job wrapper (WN) checks the agreement status when ACTIVE – job wrapper transfers output to SE_ID  Case 2: user wants to replicate some output files JDL contains OutputSE After the production of the output files, the job waits until the agreement ID status is ACTIVE then the job transfers files