1. EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org www.glite.org The gLite middleware distribution OSG Consortium Meeting Seattle, 21-23 August 2006

2. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 2 Outline Background and approach adopted Architecture Software process Status Summary

3. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 3 Background & Approach gLite –Exploit experience and existing components from VDT (Condor, Globus), EDG/LCG, AliEn, and others –Develop a lightweight stack of generic middleware useful to EGEE applications (HEP and Biomedics are pilot applications).  Pluggable components – cater for different implementations  Follow SOA approach, WS-I compliant where possible –Focus is on re-engineering and hardening –Business friendly open source license  Plan to switch to Apache-2

4. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 4 Service Oriented Architecture gLite follows a Service Oriented Architecture –Facilitate interoperability among Grid services –Allow easier compliance with upcoming standards –The services work together in a concerted way but can also be deployed and used independently, allowing their exploitation in different contexts Services communicate through the exchange of messages –Slowly moving to WS-* interfaces –Still missing a real standard. Many WS-* specifications –Activity inside GGF-GIN GINGIN

5. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 5 Middleware structure Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory Foundation Grid Middleware will be deployed on the EGEE infrastructure –Must be complete and robust –Should allow interoperation with other major grid infrastructures –Should not assume the use of Higher-Level Grid Services Foundation Grid Middleware Security model and infrastructure Computing (CE) and Storage Elements (SE) Accounting Information and Monitoring Higher-Level Grid Services Workload Management Replica Management Visualization Workflow Grid Economies... Applications

6. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 6 Workload ManagementData Management Security Information & Monitoring Access gLite Grid Middleware Services API Computing Element Workload Management Metadata Catalog Storage Element Data Movement File & Replica Catalog Authorization Authentication Information & Monitoring Application Monitoring Auditing Job Provenance Package Manager CLI Accounting Site Proxy Overview paper http://doc.cern.ch//archive/electronic/egee/tr/egee-tr-2006-001.pdf

7. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 7 Grid foundation: Accounting Resource usage by VO, group or single user –Resource metering: sensors running on resources to determine usage –Pricing policies: associate a cost to resource usage  if enabled allowed market- based resource brokering –privacy: access to accounting data granted only to authorized people (user, provider, VO manager) Basic functionality in APEL, full functionality in DGAS

8. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 8 Grid foundation: Computing Element The CE software accepts batch jobs (and job control requests) through a gatekeeper  LCG-CE (GT2 GRAM + GSI-enabled Condor)  gLite-CE (GSI-enabled Condor-C)  CREAM (WS-I based interface) performs the necessary AAA operations and map to a local user  through LCAS/LCMAPS and the GRAM or glexec passes the job to a layer that interacts with the local resource manager  BLAH monitors the status of the jobs and reports it to the client  Condor  CEMon (in CREAM)  Web service interface to the CE info Client LRMS WN Monitoring Computing Element AAA and local mapping Job Controller Grid Site

9. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 9 Grid foundation: Storage Element Site File Name (SFN): identifies a Storage Element and the logical name of the file inside it Physical File Name (PFN): argument of file open Storage Resource Manager (SRM) –hides the storage system implementation (disk or active tape) –checks the access rights to the storage system and the files –translates SFNs to PFNs –disk-based: DPM, dCache; tape-based: Castor, dCache File I/O: posix-like access from local nodes or the grid  GFAL

10. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 10 High Level Services: EDS Encrypted Data Storage –encrypt and decrypt data on-the-fly –Key-store: Hydra  N instances: at least M (<N) need to be available for decryption fault tolerance and security Demonstrated with the SRM- DICOM demo at EGEE Pisa conference (Oct’05) Will be GFAL Will be LFC Will be DPM (now d-Cache)

11. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 11 High Level Services: Workload Manag. Resource brokering, workflow management, I/O data management  Web Service interface: WMProxy –Task Queue: keep non matched jobs –Information SuperMarket: optimized cache of information system –Match Maker: assigns jobs to resources according to user requirements –Job submission & monitoring  Condor-G  ICE (to CREAM) –External interactions:  Information System  Data Catalogs  Logging&Bookkeeping  Policy Management system (G-PBox)

12. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 12 High Level Services: Job Information Logging and Bookkeeping service –Tracks jobs during their lifetime (in terms of events) Job Provenance stores long term job information –Supports job rerun

13. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 13 SA3 Testing & Certification Functional Tests Testbed Deployment gLite Software Process JRA1 Development Software Error Fixing SA3 Integration Deployment Packages Integration Tests Installation Guide, Release Notes, etc SA1 Pre- Production Scalability Tests Pre-Production Deployment Fail Pass SA1 Production Infrastructure Release Problem Serious problem Directives

14. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 14 gLite Software Process Technical Coordination Group (TCG) –gathers & prioritizes user requirements from HEP, biomed, (industry), sites –gLite development is client-driven! Software from EGEE-JRA1 and other projects –JRA1 preview testbed (currently being set up)  early exposure to users of “uncertified” components SA3 Integration Team –Ensures components are deployable and work –Deployment Modules implemented high-level gLite node types  (WMS, CE, R-GMA Server, VOMS Server, FTS, etc) –Build system now spun off into the ETICS project (Jan 2006) SA3 Certification Team –Merge of the JRA1 testing and SA1 certification teams –Dedicated testbed; test release candidates and patches –Develop test suites SA1 Pre-Production System –Scale tests by users

15. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 15 gLite status Convergence of LCG 2.7.0 and gLite 1.5.0 in spring 2006 –continuity on the production infrastructure ensured usability by experiments –new features from gLite 1.5.0 Current activities –Improve usability, efficiency and performance –Migration to VDT 1.3.11 (GT4 pre-WS) –Support for Scientific Linux 4 and 64-bit  Support for other platforms will follow –New data management components for Biomed applications on the production infrastructure –Certify new components (CREAM, Job Provenance, GPBox,...) Interoperation with other projects and adherence to standards Open source (Apache) license LCG-2 prototyping product 2004 2005 product gLite 2006 gLite 3.0

16. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 16 Summary gLite 3 is an important milestone in EGEE program –New components from gLite 1.5 being deployed for the first time on the Production Infrastructure  Address requirements in terms of functionality and scalability  Components deployed for the first time need extensive testing! –New organization in EGEE II  more controlled software process  development is client driven (TCG) Development is continuing to provide increased robustness, usability and functionality Collaboration with other projects for interoperability and definition/adoption of international standards

17. Enabling Grids for E-sciencE EGEE-II INFSO-RI-031688 OSG Consortium Meeting - Seattle - 21-23 August 2006 17 www.glite.org