1. Introduction to the Grid: technologies and projects Oxana Smirnova Lund University October 28, 2003, Košice

2. 2003-10-28 oxana.smirnova@hep.lu.se 2 Outlook Information Technology developments Grid solutions High Energy Physics challenges Development and deployment projects

3. 2003-10-28 oxana.smirnova@hep.lu.se 3 IT progress: some facts Network vs. computer performance:  Computer speed doubles every 18 months  Network speed doubles every 9 months 1986 to 2000:  Computers: 500 times faster  Networks: 340000 times faster 2001 to 2010 (projected):  Computers: 60 times faster  Networks: 4000 times faster Slide adapted from the Globus Alliance Bottom line: CPUs are fast enough; networks are very fast – gotta make use of it!

4. 2003-10-28 oxana.smirnova@hep.lu.se 4 The Grid Paradigm Distributed supercomputer, based on commodity PCs and fast WAN Access to the great variety of resources by a single pass – certificate A possibility to manage distributed data in a synchronous manner (e.g., LHC data analysis) A new commodity Supercomputer Workstation PC Farm The Grid Drainage Water Electricity Internet Grid Radio/TV

5. 2003-10-28 oxana.smirnova@hep.lu.se 5 Wider scope: a Grid System A Grid system is a collection of distributed resources connected by a network Examples of Distributed Resources: Desktop Handheld hosts Devices with embedded processing resources such as digital cameras and phones Tera-scale supercomputers Slide adapted from A.Grimshaw

6. 2003-10-28 oxana.smirnova@hep.lu.se 6 Characteristics of a generic Grid system Numerous Resources Ownership by Mutually Distrustful Organizations & Individuals Potentially Faulty Resources Different Security Requirements & Policies Required Resources are Heterogeneous Geographically Separated Different Resource Management Policies Connected by Heterogeneous, Multi-Level Networks Slide adapted from A.Grimshaw

7. 2003-10-28 oxana.smirnova@hep.lu.se 7 Grid paradigm is overloaded Desktop Cycle Aggregation Desktop only United Devices, Entropia, Data Synapse Cluster & Departmental “Grids” Single owner, platform, domain, file system and location SUN SGE, Platform LSF, PBS Enterprise “Grids” Single enterprise; multiple owners, platforms, domains, file systems, locations, and security policies SUN SGE EE, Platform Multicluster Global Grids Multiple enterprises, owners, platforms, domains, file systems, locations, and security policies Legion, Avaki, Globus Graph borrowed from A.Grimshaw WARNING! Not everything that has “G” in the name is Grid! (SGE, Oracle 10g, Condor-G etc)

8. 2003-10-28 oxana.smirnova@hep.lu.se 8 Globus: the toolkit provider The first and only provider of a Grid toolkit (libraries and API)  An academic research project in USA and now Europe  Free software, open code  Supports Grid testbeds since late 90’s Grid features: Heterogeneous Non-interactive Single logon Optimized file transfer protocol Information schema To do: Global resource management Data management User management, accounting To do: Global resource management Data management User management, accounting

9. 2003-10-28 oxana.smirnova@hep.lu.se 9 Gatekeeper (factory) Reporter (registry + discovery) User process #2 Proxy #2 Create process Register User process #1 Proxy Authenticate & create proxy credential GSI (Grid Security Infrastructure) Reliable remote invocation GRAM (Grid Resource Allocation & Management) The Globus Toolkit v2 in One Slide Grid protocols (GSI, GRAM, …) enable resource sharing within virtual organizations; toolkit provides reference implementation ( = Globus Toolkit services) l Protocols (and APIs) enable other tools and services for membership, discovery, data management, workflow, … Other service (e.g. GridFTP) Other GSI- authenticated remote service requests GIIS: Grid Information Index Server (discovery) MDS-2 (Monitoring and Discovery Service) Soft state registration; enquiry Slide adapted from the Globus Alliance

10. 2003-10-28 oxana.smirnova@hep.lu.se 10 Globus-Based Grid Tools & Applications Data Grids  Distributed management of large quantities of data: physics, astronomy, engineering High-throughput computing  Coordinated use of many computers Collaborative environments  Authentication, resource discovery, and resource access Portals  Thin client access to remote resources & services And combinations of the above Slide adapted from the Globus Alliance

11. 2003-10-28 oxana.smirnova@hep.lu.se 11 Some architectural thoughts Storage User Interface Information Server Data location server Workload manager Information Server

12. 2003-10-28 oxana.smirnova@hep.lu.se 12 Who needs Grid: High Energy Physics challenges Data-intensive tasks  Large datasets, large files  Lengthy processing times  Large memory consumption  High throughput is necessary Very distributed user base  Distributed computing resources of modest size  Produced and processed data are hence distributed, too  Issues of coordination, synchronization and authorization are outstanding HEP is by no means unique in its demands, but they are first, they are many, and they badly need it

13. 2003-10-28 oxana.smirnova@hep.lu.se 13 Experiment-Grid interaction Experiment Grid Task Input DB Output DB MSS Paper Job Description Information System Resource Broker Resources CPU Disk Monitoring & control Replica Location

14. 2003-10-28 oxana.smirnova@hep.lu.se 14 HEP-related Grid projects European projects US projects Many national, regional Grid projects -- GridPP(UK), INFN-grid(I), NorduGrid, Dutch Grid, … The Virtual Data Toolkit (VDT) The DataGRID Toolkit Slide adapted from Les Robertson

15. 2003-10-28 oxana.smirnova@hep.lu.se 15 Related Grid projects Other Grid-related projects do not develop Open Source-like (i.e., free) software/middleware, as of today  Most notably, Legion/Avaki: Globus competitor, widely used by businesses  Entropia: like SETI@Home  IBM, Platform: Globus-based  Sun Grid Engine EE: enterprise Grids 20002001200220032004200520062007200820092010 LCG EDGEGEE GriPhyN, PPDGVDT CROSSGRID DataTAG NorduGrid GlobusGT2GT3 OGSA

16. 2003-10-28 oxana.smirnova@hep.lu.se 16 What Grid can do today Simplest Grid: users access distributed resources using a single certificate More complex Grid: users’ tasks are distributed between different resources by a broker Even more complex Grid: not only tasks, but massive amounts of data are also distributed and managed (not quite there yet, only prototypes ??? Broker(s) ??? Broker(s) MSS SE MSS ???

17. 2003-10-28 oxana.smirnova@hep.lu.se 17 What is missing Common policies, or ways of mutually respecting such Grid accounting systems and Grid economy Serious security solutions; role-based access control Full-blown distributed data management systems Tools and methods for system-wide applications environment deployment STANDARDS!

18. 2003-10-28 oxana.smirnova@hep.lu.se 18 The Grid or many Grids? Globus Toolkit 2 is a basis for great many Grid solutions  Which use some common tools and utilities: GSI, GridFTP  But they also differ a lot, architecturally and technologically  There are several non-interoperable GT2-based Grid systems! No satisfactory ready-made solutions  developers invent their own Being financed from different sources, developers and users are not always encouraged to adopt rival project’s solution Instead of “How should I use Grid?”, users ask “Which Grid should I use?” Grid standards body: Global Grid Forum (GGF)  Heavily oriented towards commercial implementations  No effective standards since 2001 Meanwhile, Globus introduced the “Open Grid Services Architecture” (OGSA)  Globus Toolkit 3 is released  Not yet used by any of the development projects  Perhaps the first set of standards endorsed by GGF

19. 2003-10-28 oxana.smirnova@hep.lu.se 19 Functionality, standardization Custom solutions 1990199520002005 Open Grid Services Arch Real standards Multiple implementations Web services, etc. Managed shared virtual systems Computer science research Globus Toolkit Defacto standard Single implementation Internet standards The emergence of Open Grid standards 2010 Slide adapted from the Globus Alliance

20. 2003-10-28 oxana.smirnova@hep.lu.se 20 Open Grid Services Architecture Standard interfaces & behaviors for distributed system management Service orientation: Grid Services, in analogy to Web Services  Web services: persistent  Grid services: transient (issues: e.g., how are they discovered?)  Extending WSDL to GSDL (work with W3C) Standard service specifications  Resource management  Data management  Workflow  Security  etc. Paves the road towards interoperability and true modularity of Grid structures

21. 2003-10-28 oxana.smirnova@hep.lu.se 21 Conclusion HEP community stirred a world-wide Grid interest  Next big thing after the dot-com?.. Despite a slow start and much hype, some real work is under way  Rather, the next big thing after the WWW ! Still, no complete solution exists  Data management?  Accounting?  Security?  Standardization? With courage and patience, we should go Grid