Web: Towards Grid Interoperability Richard Boardman, Stephen Crouch, Hugo Mills, Steven Newhouse, Juri Papay and the OMII-UK Team All Hands Meeting 11/09/07

Web: Contents Introduction & background Interoperability and standards o Job Submission o Implementations Build and test Job brokering using job submission standards

Web: Introduction: What is the Grid? The grid – many definitions! “Grid computing offers a model for solving massive computational problems by making use of the unused CPU cycles of large numbers of disparate, often desktop, computers treated as a virtual cluster embedded in a distributed telecommunications infrastructure” – Wikipedia An infrastructure for “coordinated resource sharing and problem solving in dynamic, multiinstitutional virtual organizations” – Foster, Kesselman, Tuecke. “A service for sharing computer power and data storage capacity over the Internet.“ – CERN (European Organisation for Nuclear Research) Differing perceptions of the Grid: o Particle physics community: massive, loosely coupled, distributed computing environment with computing capability, bandwidth and storage. o Bio-informaticians: global virtual federated database of experimental data, research papers and laboratory records. Suggested common view: o The Grid provides secure virtualisation of resources and enables collaboration and establishment of virtual organisations.

Web: Background Many Grid infrastructures have been developed, but, traditionally, with little interoperability: o Policies governing access/use of distributed resources o Lack of adherence to emerging common standards Interoperability offers huge benefits for categories of users within user community: o e-Infrastructure providers: easier deployment/management of software distributions o e-Science users: freedom to choose services deployed in different Grids; based on functionality, not deployed on a particular Grid o e-Science application developers: portability of apps across multiple Grids to increase uptake

Web: Standards: the key to Interoperability Adoption of common standards strongly supported and implemented by OMII-Europe and GIN (Grid Interoperability Now) for: o Job Submission, Accounting, Virtual Organisation Management… o Standards from: OGF, OASIS, W3C, DMTF… o Across platforms: EGEE, Globus, UNICORE & others Will focus on Job Submission (OGSA-BES, JSDL) standards

Web: Emerging Job Submission Standards Two key standards for two key elements: o The Basic Execution Service interface (OGSA-BES) Simplified version of OGSA-EMS (Execution Management Service) Handles basic job lifecycle management Defines simple (but extendable) job state model –Pending, running, cancelled, failed or finished o The Job Submission Description Language (JSDL) Specify job executable, data staging and resource requirements

Web: Summary of OGSA-BES BES-Management Port-type StopAcceptingNewActivitiesRequest that the BES stop accepting new activities StartAcceptingNewActivitiesRequest that the BES start accepting new activities BES-Factory Port-type CreateActivityRequest the creation of a new activity GetActivityStatusesRequest the status of a set of activities TerminateActivitiesRequest that a set of activities be terminated GetActivityDocumentsRequest the JSDL documents for a set of activities GetFactoryAttributesDocumentRequest XML document containing BES properties BES-Activity Port-type (optional) GetStatusRequest the status of an activity TerminateRequest that an activity be terminated GetDocumentRequest the JSDL document for an activity GetActivityAttributesDocumentRequest XML document containing activity properties

Web: Interface/Interaction Standards not Enough BES & JSDL alone not enough for real interoperability o JSDL is extensible o Differing security models across Grid infrastructures HPC-Profile proposes Grid interoperability through: o Restricted OGF Job Submission Description Language (JSDL) o OGF OGSA Basic Execution Service (BES) o WS-I Basic Profile In addition, an agreed security framework between participants o Via HTTPS transport (server offers certificate) & username/password (client) for user authentication OMII-UK has implemented the HPC-Profile within: o GridSAM – funded by OMII-UK, first to adopt BES o CROWN – with OMII-Europe, in collaboration with Beihang University, China

Web: Build and Test To gain confidence of compliance with interoperability standard, test against other standards-compliant infrastructures o Test multiple BES/JSDL clients against multiple service endpoints o Mechanistic process; automation advantageous ETICS (CERN) provides test automation framework o e-Infrastructure for Testing, Integration and Configuration of S/W o Leverages Metronome (formerly NMI Build & Test) across Condor cluster o Controls management of software builds and testing o Create project configurations, maintain historical records ETICS deployed to enable automated compliance testing

Web: NMI Build and Test

Web: ETICS

Web: Build and Test BES method sequence for testing (core functionality): o Get attributes document o Create job o Query job status o Show job output o Get job’s JSDL document o Terminate job Scenarios implemented in ETICS to test interoperability o Testing across HPC-Profile endpoints (with simple config) o Test OMII-UK client component against service

Web: Job Brokering using the CROWN Scheduler CROWN Grid developed by Beihang University With OMII-Europe as part of Component Exchange activity with OMII China: o Identified CROWN Meta Scheduler o Integrated into OMII release (interoperation) Usage of OMII-UK Grimoires service registry o Coordinated implementation of BES interface to the Scheduler

Web: CROWN BES Scheduler BES Client Registry submit monitor submit monitor submit monitor CROWN Scheduler Supported Platform 2 jobs, different requirements Delegate BES resource selection by submitting both jobs to Scheduler Monitor each until completion BES 1. request resources 3. monitor Grimoires 2. submit 3. monitor 2 jobs, different requirements Identify 2 appropriate BES instances Submit to both

Web: CROWN BES Scheduler Demo

Web: Conclusions First phase of projects dealt with infrastructures, test beds and application software o Led to greater understanding of key issues Next phase concentrates on interoperability and providing solutions-based approaches Interoperability offers key benefits to the community: o Ease of management o Choice o Simplicity of implementation Have illustrated a process of standards adoption, build and compliance testing, and usage within a scheduling application