INFSO-RI Enabling Grids for E-sciencE Concepts of grid computing Guy Warner NeSC Training Team

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05 2 Acknowledgements This talk was prepared by Mike Mineter of NeSC and includes slides from previous tutorials and talks delivered by: –Dave Berry, Richard Hopkins, Guy Warner (National e-Science Centre) –the EDG training team –Ian Foster, Argonne National Laboratories –Jeffrey Grethe, SDSC –EGEE colleagues –Mark Baker, The Distributed Systems Group, University of Portsmouth, Talks at 3 rd EGEE conference by –Kyriakos Baxevanidis,Deputy Head,Unit of Research Infrastructures,European Commission, DG INFSO – Dr Spyros Konidaris, European Commission – DG INFSO

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05 3 GRIDMIDDLEWAREGRIDMIDDLEWARE Visualising Workstation Mobile Access Supercomputer, PC-Cluster Data-storage, Sensors, Experiments Internet, networks The Grid Metaphor

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05 4 The grid vision The grid vision is of “Virtual computing” (+ information services to locate computation, storage resources) –Compare: The web: “virtual documents” (+ search engine to locate them) MOTIVATION: collaboration through sharing resources (and expertise) to expand horizons of –Research –Commerce – engineering, … “the knowledge economy” –Public service – health, environment,…

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05 5 “A grid” The initial vision: “The Grid” The present reality: Many “grids” Each grid is an infrastructure enabling one or more “virtual organisations” to share computing resources What’s a VO? –People in different organisations seeking to cooperate and share resources across their organisational boundaries Why establish a Grid? –Share data –Pool computers –Collaborate Institute A VO Institute CInstitute BInstitute D

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05 6 The Single Computer The Operating System enables easy use of –Input devices –Processor –Disks –Display –Any other attached devices Disks, Processor, Memory, … Application Software Operating System

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05 7 Resources on a Local Area Network Resources connected by a LAN Application Software Operating System on each computer Middleware for sharing computers, servers, printers, … User just perceives “shared resources”, with no regard to location in the organisation: - Authenticated by username / password - Authorised to use own files,…

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05 8 Resources on a grid Resources connected by internet Application Software Operating System on each resource Grid Middleware on each resource Grid Middleware: “collective services” Interface between app. and grid

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July 05 9 A Grid THE INTERNE T THE INTERNE T Grid Middleware on each shared Resource Local Area Networks –Connected by Internet –Data Storage –(Usually) batch jobs on pools of processors Users join VO’s Virtual organisation negotiates with sites to agree access to resources Distributed services (both people and middleware) enable the grid, allow single sign- on

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July Grid Middleware on each shared Resource Local Area Networks –Connected by Internet –Data Storage –(Usually) batch jobs on pools of processors Users join VO’s Virtual organisation negotiates with sites to agree access to resources Distributed services (both people and middleware) enable the grid, allow single sign- on A Grid (Printable Version) THE INTERNE T THE INTERNE T

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July What characterises a grid? Co-ordinated resource sharing –No centralised point of control –Different administrative domains. Standard, open, general-purpose protocols and interfaces –NOT specific to an application –EGEE, NGS support multiple VO’s Delivering non-trivial qualities of service –Co-ordinated to deliver combined services, greater than sum of the individual components

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July The components of a Grid Resources –networking, computers, storage, data, instruments, … Grid Middleware –the “operating system of the grid” Operations infrastructure –Run enabling services (people + software) Virtual Organization management –Procedures for gaining access to resources

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July Key concepts Virtual organisation: people and resources collaborating - across admin, organisational boundaries Single sign-on –I connect to one machine – some sort of “digital credential” is passed on to any other resource I use, basis of:  Authentication: How do I identify myself to a resource without username/password for each resource I use?  Authorisation: what can I do? Determined by My membership of VO VO negotiations with resource providers Grid middleware runs on each resource User just perceives “shared resources” with no concern for location or owning organisation

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July The first driver: e-Science What is e-Science? Collaborative science that is made possible by the sharing across the Internet of resources (data, instruments, computation, people’s expertise...) –Often very compute intensive –Often very data intensive (both creating new data and accessing very large data collections) – data deluges from new technologies –Crosses organisational boundaries

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July Curation, discovery, re- use of knowledge e-Research The expanding horizons of grids e-Science

Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July The National Grid Service (NGS) NGS is a production service –Therefore cannot include latest research prototypes! –ETF recommends what should be deployed Core sites provide computation and also data services NGS is evolving –OMII, EGEE, Globus Alliance all have m/w under assessment by the ETF for the NGS Selected, deployed middleware currently provides “low-level” tools –New deployments will follow soon –New sites and resources being added !

Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July U of A HPCxHPCx UofD GOSC NGS Core Nodes: Host core services, coordinate integration, deployment and support +free to access resources for all VOs. Monitored interfaces + services NGS Partner Sites: Integrated with NGS, some services/resources available for all VOs Monitored interfaces + services NGS Affiliated Sites: Integrated with NGS, support for some VO’s Monitored interfaces (+security etc.) RAL Oxford Leeds Man. CSARCSAR U of B U of C Commercial Provider PSRE BRISTOL CARDIFF

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July EGEE – building e-infrastructure EGEE is building a large-scale production grid service to: Underpin research, technology and public service Link with and build on national, regional and international initiatives Foster international cooperation both in the creation and the use of the e- infrastructure Network infrastructure & Resource centres Operations, Support and training Collaboration Pan-European Grid

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July EGEE Communities Initially supported two communities: High Energy Physics and Bioinformatics –Most VO’s linked to a particular experiment Additional Communities have since been added: –Geophysics –Earth Observation –Chemistry Working with other communities –E.g. Digital Libraries Pilot Added

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July … then where are we now? If “The Grid” vision leads us here…

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July Grids: where are we now? Many key concepts identified and known Many grid projects have tested, and benefit from, these Major efforts now on establishing: –Standards (a slow process) (e.g. Global Grid Forum, ) –Production Grids for multiple VO’s  “Production” = Reliable, sustainable, with commitments to quality of service In Europe, EGEE In UK, National Grid Service In US, Teragrid  One stack of middleware that serves many research (and other!!!) communities  Operational procedures and services (people!, policy,..) –New user communities … whilst research & development continues

Enabling Grids for E-sciencE INFSO-RI Multimodal Behavioural Data and e-Collaboration, NeSC, Edinburgh, 14 July Summary of grid computing concepts Flexible collaboration across multiple administrative domains – sharing data, computers, instruments, application software,.. Single sign-on to resources in multiple organisations –Authorisation, authentication Need for people-services as well as middleware services –credential authorities, VO managers, support Drives are towards –Production services (reliable, sustainable,… – against which research projects can plan with confidence)  In Europe, EGEE  In UK, National Grid Service –Standards –Empowering new user communities