EGEE-II INFSO-RI Enabling Grids for E-sciencE Overview of gLite, the EGEE middleware Mike Mineter Training Outreach Education National e-Science Centre, Edinburgh, UK With thanks to EGEE colleagues for many of these slides

Enabling Grids for E-sciencE EGEE-II INFSO-RI Contents The major components of a Grid gLite components Overview of the remaining sessions of the course Sources of further information

Enabling Grids for E-sciencE EGEE-II INFSO-RI INTERNET Virtual organisations bring and/or negotiate access to resources Grid middleware runs on each shared resource Provides –Data services –Computation services –Single sign-on Distributed services (both people and middleware) enable the grid Typical current grid At each site that provides computation: Local resource management system (= batch queue) PBS … EGEE term: queue is a “Computing element” JOBS RUN IN BATCH QUEUE

Enabling Grids for E-sciencE EGEE-II INFSO-RI Authentication Users in many locations and organisations Computing clusters,…Network resourcesData storage Operating systemLocal schedulerFile system Hardware System software HPSS, CASTOR… NFS, … PBS, Condor, LSF,… Resources in many locations and organisations GRID SERVICES Access services (“user interface”) : logon, upload credentials, run m/w commands map user’s credential to local user id / account Build on Grid Security Infrastructure “credential” ~ a file with your identity, digitally signed by recognised authority

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workload management Users Compute elements Network resources Data storage Resources How do I run a job on a compute element (CE) ? (CE =batch queue) Tools that provide abstraction: Submit job Monitor job Get output copy files to and from CE’s Without you knowing about CEs

Enabling Grids for E-sciencE EGEE-II INFSO-RI Information service (IS) Users Compute elements Network resources Data storage Resources How do I know which CE could run my job? Which is free? Information Service (IS): Resources send updates to IS Grid services (or users) query IS before running jobs

Enabling Grids for E-sciencE EGEE-II INFSO-RI File management Users Compute elements Network resources Data storage Resources My data are in files, and I’ve terabytes Storage Transfer Replica management Metadata service Our data are in files, and I’ve terabytes We’ve big files to manage and share… EGEE data: primarily file-based

Enabling Grids for E-sciencE EGEE-II INFSO-RI Authorisation Users in many locations and organisations Computing clusters,…Network resourcesData storage Operating systemLocal schedulerFile system Hardware System software HPSS, CASTOR… NFS, … PBS, Condor, LSF,… Resources in many locations and organisations Virtual Organisation Membership Services join a recognised VO VO negotiates access Do we need to negotiate an account on every CE?!

Enabling Grids for E-sciencE EGEE-II INFSO-RI Major gLite components

Enabling Grids for E-sciencE EGEE-II INFSO-RI User Interface (UI) User Interface (UI):The place where users logon to the Grid Computing Element (CE) Computing Element (CE): A batch queue on a site’s computers where the user’s job is executed Storage Element (SE) Storage Element (SE): provides (large-scale) storage for files Workload Management System (WMS) Workload Management System (WMS): Matches the user requirements with the available resources resources on the Grid Main components Information System Information System: Characteristics and status of CE and SE VO Membership System VO Membership System manages VOs – basis for authorisation

Enabling Grids for E-sciencE EGEE-II INFSO-RI User Interface (UI) User Interface (UI):The place where users logon to the Grid Computing Element (CE) Computing Element (CE): A batch queue on a site’s computers where the user’s job is executed Storage Element (SE) Storage Element (SE): provides (large-scale) storage for files Workload Management System (WMS) Workload Management System (WMS): Matches the user requirements with the available resources resources on the Grid Main components Information System Information System: Characteristics and status of CE and SE VO Membership System VO Membership System manages VOs – basis for authorisation

Enabling Grids for E-sciencE EGEE-II INFSO-RI Current production middleware ReplicaCatalogue Logging & Book-keeping Workload Management StorageElementComputingElement InformationService Job Status DataSets info Author. &Authen. Job Submit Event Job Query Job Status Input “sandbox” Input “sandbox” + Job Info Output “sandbox” Publish SE & CE info “User interface”

Enabling Grids for E-sciencE EGEE-II INFSO-RI Who provides the services?! ServiceProviderNote Access service User / institute / VOComputer with client software Workload management VO / Grid operationsmost-used (not only) way to run jobs Information System Information System:Grid operations Computing Element (CE) VOs - EGEE does not fund CEs Scalability requires that VOs provide resources to match average need Storage Element (SE) VOs Virtual Organisation Membership System (VOMS) VO / Grid operationsVO and its server need to be accepted by resources “VO”: virtual organisation

Enabling Grids for E-sciencE EGEE-II INFSO-RI gLite Middleware structure Access for applications to: –Higher-level Grid Services –Foundation Grid Middleware Higher-Level Grid Services to help build a community’s infrastructure - options and alternatives exist –Recommended External Software Packages for Egee CommuniTies –Current RESPECT tools:  GridWay  P-GRADE Portal –  “Grid software” menu Foundation gLite Middleware deployed on the EGEE infrastructure –Should not assume the use of Higher-Level Grid Services Foundation gLite Middleware Security model and infrastructure Computing (CE) and Storage Elements (SE) Accounting Information and Monitoring Higher-Level Grid Services Some from gLite: e.g. Workload Management System Some from other developers alterna - tive brokers (GridWay) portals and workflow (GENIUS, P-GRADE, Taverna + MOTEUR).. Applications

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application families Simulation –Large number of similar, independent jobs – parameter study Bulk Processing –Widely-distributed input data, Sophisticated data management Workflow –Complex dependencies between individual tasks Legacy Applications –Licenses: control access to software on the grid –No recompilation  no direct use of grid APIs Parallel Jobs –Many CPUs needed simultaneously, Use of MPI libraries –Limited support in gLite: MPI configuration is not uniform Responsive Apps. –Short response time –No real support in gLite  Interactive Grid FP6 project

Enabling Grids for E-sciencE EGEE-II INFSO-RI Supported platforms gLite is presently supporting Scientific Linux 3 and 4. Scientific Linux CERN 4 is a Linux distribution build within the framework of Scientific Linux which in turn is rebuilt from the freely available Red Hat Enterprise Linux 4 product sources under terms and conditions of the Red Hat EULA. Scientific Linux CERN is built to integrate into the CERN computing environment but it is not a site-specific productScientific LinuxRed Hat Enterprise Linux 4

Enabling Grids for E-sciencE EGEE-II INFSO-RI Course overview

Enabling Grids for E-sciencE EGEE-II INFSO-RI Course structure Security – its all built on this! Authentication: How I communicate my identity to services I want to use, and how they prove their identity – based on “GSI” Delegation: how I can delegate a service to act on my behalf in using another service Authentication: how VOMS communicates the VO and groups in a VO that I belong to, so services can make decisions about what I can do 2. Information services What resources exist on the Grid I am using?

Enabling Grids for E-sciencE EGEE-II INFSO-RI Workload management Jobs run in batch mode How do I submit, monitor, retrieve output? More complex types of jobs – that run many sub-jobs at the same time using many CEs at once 4. File management How to store, transfer and access files on “storage elements” How to share these with members of my VO Metadata: we may have many files – how to hold information about the file contents so we can access what we need

Enabling Grids for E-sciencE EGEE-II INFSO-RI In practicals… We will be using the GENIUS portal From browser running the major gLite services Application developers will usually use the command line or API interfaces – these are used in GENIUS. We will show you the command line, although its not being used in this course

Enabling Grids for E-sciencE EGEE-II INFSO-RI Summary EGEE is creating the “grid layer” in e-Infrastructure for research, public service and industry gLite is the middleware that EGEE deploys The rest of the course explores the gLite components Above all….. Ask questions so together we can explore how this can serve your interests.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Further information, references EGEE – gLite middleware – gLite manuals, documentation – (gLite user guide) Recommended External Software Packages for Egee CommuniTies (RESPECT) –

Enabling Grids for E-sciencE EGEE-II INFSO-RI Summary EGEE is running the largest multi-VO grid in the world! Key concepts for EGEE –Sustainability – planning for the long-term –Production quality EGEE’s middleware: gLite