1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite Slides by: Erwin Laure EGEE Deputy Middleware Manager

2. Enabling Grids for E-sciencE EGEE Objectives –consistent, robust and secure service grid infrastructure –improving and maintaining the middleware –attracting new resources and users from industry as well as science Structure –70 leading institutions in 27 countries, federated in regional Grids –leveraging national and regional grid activities worldwide –funded by the EU with ~32 M Euros for first 2 years starting 1st April 2004 –proposal for 2 nd phase (April 2006-March 2008) currently being negotiated

3. Enabling Grids for E-sciencE Grid Middleware Key success factors for production quality software: Strict software process –Use industry standard software engineering methods Software configuration management, version control, defect tracking, automatic build system, … Conservative in what software to use –Avoid cutting-edge software Deployment on over 100 sites cannot assume a homogenous environment – middleware needs to work with many underlying software flavors –Avoid evolving standards Evolving standards change quickly (and sometime significantly cf. OGSI vs. WSRF) – impossible to keep pace on over 100 sites You will not develop and deploy your PhD project on a production Grid infrastructure There is a long (and tedious) path from prototypes to production

4. Enabling Grids for E-sciencE EGEE Middleware: gLite gLite –Exploit experience and existing components from VDT (Condor, Globus), EDG/LCG, AliEn, and others –Develop a lightweight stack of generic middleware useful to EGEE applications (HEP and Biomedics are pilot applications). Should eventually deploy dynamically (e.g. as a globus job) Pluggable components – cater for different implementations –Focus is on re-engineering and hardening

5. Enabling Grids for E-sciencE gLite Software Process DevelopmentIntegrationTesting Software Code Deployment Packages Integration Tests FailPass Fix Functional Tests Testbed Deployment Fail Pass Installation Guide, Release Notes, etc

6. Enabling Grids for E-sciencE gLite Grid Middleware Services API Access Workload Mgmt Services Computing Element Workload Management Metadata Catalog Data Management Storage Element Data Movement File & Replica Catalog Authorization Security Services Authentication Information & Monitoring Information & Monitoring Services Application Monitoring Connectivity Accounting Auditing Job Provenance Package Manager CLI

7. Enabling Grids for E-sciencE gLite Key Concepts Centered around VOs –Its ultimately the VO who gets resources allocated and need to decide how to best use them (share them among the VO users) Distinguish between infrastructure and VO services Infrastructure services –Operated and trusted by the resource administrator –Implement site policies Including what share of the resources are allocated to a VO –Provide the required security, auditing, and accounting –Grid and standard services E.g. batch system, gatekeeper, gridFTP, …

8. Enabling Grids for E-sciencE gLite Key Concepts VO services –Implement intra-VO policies Scheduling, priorities, etc. –Managed and operated by a VO Typically by sites on behalf of VOs A service instance may serve multiple VOs –Currently mostly higher level services Resource brokers, catalogs, … –There is the need of deploying VO services closer to the resource Better information about the resource and better control about the resource Downside: more and more services to be deployed at the sites – see discussion later on

9. Enabling Grids for E-sciencE Service Oriented Architecture Guiding Principles Interoperability Portability Modularity Scalability Web Services Building on existing components in a lightweight manner AliEnLCG Condor GlobusSRM...

10. Enabling Grids for E-sciencE Security: Basic Concepts Grid security is based on X.509 PKI infrastructure –Certificate Authorities (CA) issue (long lived) certificates identifying individuals (much like a passport) –Trust between CAs and sites is established (offline) –User identification is done by using (short lived) proxies of their certificates Proxies can –Be delegated to a service such that it can act on the users behalf –Include additional attributes (like VO information via the VO Membership Service VOMS) –Be stored in an external proxy store (myProxy) –Be renewed (in case they are about to expire) Standard TLS (like ssh) and MLS (like WS-Sec) is used –MLS has performance and support issues!

11. Enabling Grids for E-sciencE Middleware Challenges: Data Management Heterogeneity –Data is stored on different storage systems using different access technologies Distribution –Data is stored in different locations – in most cases there is no shared file system or common namespace –Data needs to be moved between different locations Different Administrative Domains –Data is stored at places you would normally have no access to –Security and auditing implications –Need common interface to storage resources Storage Resource Manager (SRM) –Need to keep track where data is stored File and Replica Catalogs –Need scheduled, reliable file transfer File transfer and placement services –Need a common security model ACLs enforcement based on Grid identities – DNs

12. Enabling Grids for E-sciencE Data Management Architecture File and Replica Catalog StorageIndex Fireman Database WMS Storage Element SRM Storage gLite I/OgridFTP File Transfer and Placement Service FTS FPS Transfer Agent Database VOMS MyProxy Get credential Store credential File I/O File namespace and Metadata mgmt File replication Proxy renewalReplica Location WSDL API

13. Enabling Grids for E-sciencE gLite Transfer services File Transfer/Placement Service (FTS,FPS) –Transfer Job Database –Exposes the Transfer Web Service Interface to which user clients talk (submit, cancel, status capability) –Has a Web Interface –Manages Catalog updates if necessary Transfer Agent –Basic Actions Get transfer jobs from Transfer Job Database Manages transfer over many channels Monitors transfer status and updates Transfer Job Database –Extensible with user-defined custom actions Retry Policy Transfer Service (glite-url-copy) –Actually performs transfer: SRM – SRM, gsiftp – SRM, gsiftp – gsiftp –Monitor capability, including gsiftp performance markers Job DB FTS/FPS WebService Transfer Agent Actions Channel glite-url-copy Web Monitor

14. Enabling Grids for E-sciencE Middleware Challenges: Workload Management Computational tasks of thousands of users need to be scheduled on the available Grid resources Grid (Meta)Scheduling consists of: –Resource Discovery/Brokering Find suitable resources –Matchmaking Assign a job to a resource that satisfies job requirements –Job execution Reliably execute the jobs and retrieve output Deal with error management Job execution requires to find the right Computing Element (computing resource) – with maybe boundary conditions (architecture, software installed, data accessible, etc.)

15. Enabling Grids for E-sciencE Workload Management Architecture

16. Enabling Grids for E-sciencE CEs & Managed Services CEs provide also a framework for deploying VO specific services VO services need to be Managed Services Ensure they dont consume more resources as allocated Provide persistency and management functions (start, stop, suspend, resume) Adhere to site security, auditing, and accounting policies All that could be done by site admins but it would be favorable to have infrastructure services taking care of that

17. Enabling Grids for E-sciencE Managed Services Architecture Authorization & configuration policies Managed Service Factory Resource Manager Management Clients Managed service factory functions Resource manager commands (Managed services) Monitoring & enforcement Managed Service Clients

18. Enabling Grids for E-sciencE More Information The EGEE Project –http://www.eu-egee.orghttp://www.eu-egee.org The LCG Project –http://cern.ch/lcghttp://cern.ch/lcg The gLite middleware –http://www.glite.orghttp://www.glite.org The Condor Project –http://www.cs.wisc.edu/condorhttp://www.cs.wisc.edu/condor The Globus Project –http://www.globus.orghttp://www.globus.org