gLite Grid Services Salma Saber (salma_saber@eri.sci.eg) ERI-Cairo, Egypt Joint EPIKH/EUMEDGRID Support event in Cairo Africa 4 – Site Admin Tutorial + Remote site inst. Egypt, 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 What is Grid? Collection of geographically distributed heterogeneous resources. Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Grid Systems & Applications aims to Integrate Virtualize Manage RESOURCEs and SERVICEs across different Vos VO – dynamic collection of multiple organizations providing coordinated resource sharing”. The gLite middleware is the set of software packages able to do this. www.glite.org Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Grid User Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Grid overview Members of the grid can dynamically be organized into multiple virtual organizations. Each of these VOs may have different policies and administrative control. All the VOs are part of a large grid and can share resources accordingly to several policies. The resources shared among VOs may be data, special hardware, processing capability, Software and licenses. Members of a grid can be part of multiple VOs at the same time. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Grid Architecture Requirements Heterogeneous (OSes, Devs, Apps.) VO Resource Sharing (Management, Security and Accounting) Resource Utilisation (Reservation, Metering, Monitoring and Logging) Job Execution (VO access, LCM, WFM, SLA) Data Services (Integration, Provisioning, Cataloguing, Metadata) Security (Authentication, Authorisation) Administrative Costs (Provisioning, Deployment and Configuration) Scalability Availability (Disaster Recovery, Fault Management) Application Specific Requirements: (EGEE: HEP, BioMed) https://savannah.cern.ch/support/?group=egeeptf Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 What is gLite? The middleware for grid computing developed by EGEE. Provides a framework for building grid applications tapping into the power of distributed computing and storage resources across the internet. gLite follows a Service Oriented Architecture. Facilitate the interoperability among Grid services. Allow easier compliance with upcoming standards. The services work together in a concerted way but can also be deployed and used independently allowing their exploitation in different contexts. Lightweight (existing) services. Easily and quickly deployable. Open source license. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 gLite Terminology Computational Resource Physical machines on which users wants to run their programs and store (or access) data files, without worrying about their physical location Job A computational task (a binary application or script) that a user wants to run on the Grid, and retrieve the results Job Submission It is the action of delegating the application to the Grid middleware for its execution. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 gLite Main components services The User Interface (UI) is the user entry point. The Workload Management System (WMS) is a set of services which are in charge of finding the suitable resources for the submitted jobs and schedule the jobs. The Logging and bookkeeping service (LB) keep track of user job execution in terms of statuses: Ready, Scheduled, Waiting, Running, Done. The Computing element (CE) is the computational resource, the entry point to a cluster or PCs handled by a job queue management system; in particular: TORQUE, PBS, LSF, CONDOR. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 gLite Main components services The Worker Nodes (WNs) are the machines where jobs are really executed. The Information System and Monitoring (IS and MON now deprecated) maintain data related to available grid resources and their health statuses. The Virtual Organization Management service (VOMS) is the way gLite improves the management of authentication and authorization to the Grid resources. The VOMS allows to their own members to define different access rights to VO’s resources. The Storage element (SE) and the File catalogue (LFC) allow to manage Grid files and offer a mechanism to locate them easily for users and jobs. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Job life cycle Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 gLite Middleware Services Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 gLite – Grid access Two possibilities: APIs or CLI Built on top of GridPortals and GUIs Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 gLite Security Authentication based on X.509 PKI infrastructure Certificate Authorities (CA) issue (long lived) certificates identifying individuals (much like a passport). Certification Authorities (CA) can generate service certificates which identify Grid services in the whole Grid. In order to reduce vulnerability, on the Grid user identification is done by using (short lived-12 hours) proxies of their certificates. The use of Proxy certificates allows the following: Delegation: in which a remote service acts on behalf of the user as long as the delegation remains valid. It is used to securely transfer user’s credentials to a service. This happens by making signed copies of the original proxy. (Single Sign On) Add additional info (Add VO specific information provided by VOMS)‏. Store a long term proxy on a secure server (MyProxy). Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 gLite Security: AutH/AhtZ Authentication The user receive a certificate from a CA. He connects to the UI via SSH. He Creates the proxy (single sign on) – All grid services will use this proxy to identify the user. Authorization The user has to subscribe to a VO (VOMS). The VO establishes the user rights. In any Grid service it will be verified if the user belongs to the VO and assigns the proper access rights to the user. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 VOMS Virtual Organization Membership Service is a service that keeps track of the members of a VO. Support MyProxy (stored proxies). VOMS Grants users authorization to access the resource at VO level. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 VOMS It extends the proxy certificate with info on VO membership, groups, roles(e.g. administrator, software engineer, student) & capabilities. Each VO has a database containing group membership, roles and capabilities information for each user. User contacts VOMS server requesting his authorization information. Server sends authorization info to the client. Client includes them in a proxy certificate. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Information System and Monitoring What? System to collect information on the state of resources. Why? To discover resources of the grid and their nature. To check for health status of resources. To provide data in order to manage the workload more efficiently. How? Monitoring and publishing fresh data on the state of resources. Who? User searching specific resources for their activity. Workload Management System. Other monitoring system Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Information System and Monitoring The IS architecture used in gLite is Berkeley Database Information Index (BDII): Stores information at VO level. Site GIIS (Grid Index Information Server): Stores information at site level. GRIS(Grid Resource Information Server): Stores information at resource level. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Information System and Monitoring Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Workload Management WMS (Workload Management System) is a set of middleware components which responsible of distribution and management of jobs across Grid resources. Two core components of WMS: WM(Workload Manager): accepts and satisfy requests for job management. (Matchmaking) is the process of assigning the best available resource. LB (Logging & Bookkeeping): keeps track of job execution in term of events: (Submitted, Running, Done,...) Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Computing Element Service that represents the computing resource that is responsible of to manage the queue of jobs to execute. CE responsible to collect accounting info. The CE may be used by a Generic Client: an end-user interacting directly with the Computing Element, or by the Workload Manager, which submits a given job to an appropriate CE found by the matchmaking process. Two job submission models : PUSH (Eager Scheduling) (jobs pushed to CE), PULL (Lazy Scheduling) (jobs coming from WMS when CE has free slots) Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Computing Element Job management through the WMS provides many benefits compared to direct job submission to the CE: The WMS can manage multiple CEs, and is able to forward jobs to the one which better satisfies a set of requirements, which can be specified as part of the job description. The WMS can be instructed to handle job failures: if a job aborts due to problems related to the execution host, the WMS can automatically resubmit it to a different CE. The WMS provides a global job tracking facility using the LB service. The WMS supports complex job types (job collections, job with dependencies) which can not be handled directly by the CEs. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Storage Element The Storage Element is the service which allows a user or an application to store data for future retrieval. To define a storage element, we need to know: Storage Resource Manager (SMR). Storage Resource Types. Transfer Protocol. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Storage Element Storage Resource Manager (SMR): It is a middleware interface application that makes standard data management operations between SEs of different resource type transparent to user. These data management operations include: File transfer. Space reservation. Renaming of files. File directory creation. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Storage Element Storage Resource Types: For relatively small SEs: Disk-based storage implementation is employed together with disk pool manager, as the storage resource manager. For bigger SEs: The mass storage system (MSS) is implemented with CASTOR (CERN Advanced STORage Manager) as the storage resource manager. For hybrids between disk pool storage and MSS, we have dCache as the storage resource manager. Transfer Protocol: To transfer of files in and out of the SE. Globus GridFTP mandatory. Others if available (https, ftp, etc). Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 LFC File Catalog keeping track of the location of files. Organize files in a logical way so that a file will be accessible from anywhere. The identification of files on the storage elements is done through the use of different identifiers: Logical File Name (LFN) An alias created by a user to refer to some item of data, e.g. “lfn:/grid/gilda/20030203/run2/track1” Globally Unique Identifier (GUID) A non-human-readable unique identifier for an item of data, e.g. “guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6” Storage URL (SURL) (or Physical File Name (PFN) )‏ The location of an actual piece of data on a storage system, e.g. “srm://grid009.ct.infn.it/dpm/ct.infn.it/gilda/output10_1” Transport URL (TURL)‏ Contains access protocol and physical port information that allow for the physical retrieval of the file, done by SRM e.g. “rfio://lxshare0209.cern.ch//data/alice/ntuples.dat” Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 LFC File Catalog while GUIDs and LFNs are used for the identification of files, storage URLs and transport URLs provide the necessary information to access and retrieve the files concerned Resolves logical filenames (LFN) to physical location of files (URL understood by SRM) and storage elements. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Metadata AMGA – Arda Metadata Grid Application Metadata is data about data. On the Grid: information about files Describe files. Locate files based on their contents through queries Metadata can describe any grid entity/object ex: JobIDs - add logging information to your jobs Monitoring of running applications: ex: ongoing results from running jobs can be published on the metadata server Simplified DB access on the grid Grid applications that needs structured data can model their data schemas as metadata. Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010

Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010 Questions Cairo, Joint EPIKH/EUMEDGRID Support Site Admin Tutorial 17.10.2010