1. Major Grid Computing Initatives Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer Science The University of Chicago

2. ARGONNE  CHICAGO Ian Foster Overview l The Grid concept l Historical background l Grid computing initiatives l Grid technology roadmap l Data Grid projects l Summary

3. ARGONNE  CHICAGO Ian Foster The Grid Concept l Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals—in the absence of central control, omniscience, trust relationships l Via investigations of u New applications that become possible when resources can be shared in a coordinated way u Protocols, algorithms, persistent infrastructure to facilitate sharing

4. ARGONNE  CHICAGO Ian Foster A Little History l Early 90s u Gigabit testbeds, metacomputing l Mid to late 90s u Early experiments (e.g., I-WAY), academic software projects (e.g., Globus), application experiments l 2000 u Major application communities emerging u Major infrastructure deployments u Clear architecture picture, rich technology base u Grid Forum: >300 people, >90 orgs, 11 countries

5. ARGONNE  CHICAGO Ian Foster Layered Grid Architecture (By Analogy to Internet Architecture) Application Fabric “Controlling things locally”: Access to, & control of, resources Connectivity “Talking to things”: communication (Internet protocols) & security Resource “Sharing single resources”: negotiating access, controlling use Collective “Managing multiple resources”: ubiquitous infrastructure services User “Specialized services”: user- or appln-specific distributed services Internet Transport Application Link Internet Protocol Architecture

6. ARGONNE  CHICAGO Ian Foster Grid Technology Base l Development of Grid protocols & services u Protocol-mediated access to remote resources u New services: e.g., resource brokering u “On the Grid” = speak Intergrid protocols u Mostly (extensions to) existing protocols l Development of Grid APIs & SDKs u Facilitate application development by supplying higher-level abstractions l The (hugely successful) model is the Internet l The Grid is not a distributed OS!

7. ARGONNE  CHICAGO Ian Foster U.S. Grid Computing Activities (Excluding Data Grid Projects) l NSF: Fundamental IT research, plus u NSF PACI program (~$3M/yr) u NEESgrid ($10M over 3 years) l DOE SC: Fundamental IT research, plus u NGI program (done), SciDAC (perhaps) l DOE DP: DISCOM ($3M/yr?) l DARPA: Parts of Quorum ($2M/yr?) l NASA: Information Power Grid (~$5M/yr) Funds [inadequate] support for research, development, deployment, operations

8. ARGONNE  CHICAGO Ian Foster Data Intensive Computing and Grids l The term “Data Grid” is often used u Unfortunate as it implies a distinct infrastructure, which it isn’t; but easy to say l Data-intensive computing shares numerous requirements with collaboration, instrumentation, computation, … l Important to exploit commonalities as very unlikely that multiple infrastructures can be maintained l Fortunately this seems easy to do!

9. ARGONNE  CHICAGO Ian Foster Emerging Data Grid Architecture Discipline-Specific Data Grid Application Coherency control, replica selection, task management, virtual data catalog, virtual data code catalog, … Replica catalog, replica management, co-allocation, certificate authorities, metadata catalogs, Access to data, access to computers, access to network performance data, … Communication, service discovery (DNS), authentication, authorization, delegation Storage systems, clusters, networks, network caches, … User Appln Collective Resource Connect Fabric

10. ARGONNE  CHICAGO Ian Foster Major Data Grid Projects l Clipper (DOE Science) u Technologies for reliable high-speed transfer l Earth System Grid (DOE Office of Science) u DG technologies, climate applications l European Data Grid (EU) u DG technologies & deployment in EU l GriPhyN (NSF ITR) u Investigation of “Virtual Data” concept l Particle Physics Data Grid (DOE Science) u DG applications for HENP experiments

11. ARGONNE  CHICAGO Ian Foster High-Level View of Earth System Grid: A Model Architecture for Data Grids Metadata Catalog Replica Catalog Tape Library Disk Cache Attribute Specification Logical Collection and Logical File Name Disk ArrayDisk Cache Application Replica Selection Multiple Locations NWS Selected Replica GridFTP commands Performance Information & Predictions Replica Location 1Replica Location 2Replica Location 3 MDS

12. ARGONNE  CHICAGO Ian Foster GriPhyN Overview (www.griphyn.org) l 5-year, $12M NSF ITR proposal to realize the concept of virtual data, via: 1) CS research on l Virtual data technologies (info models, management of virtual data software, etc.) l Request planning and scheduling (including policy representation and enforcement) l Task execution (including agent computing, fault management, etc.) 2) Development of Virtual Data Toolkit (VDT) 3) Applications: ATLAS, CMS, LIGO, SDSS l PIs=Avery (Florida), Foster (Chicago)

13. ARGONNE  CHICAGO Ian Foster The Petascale Virtual Data Grid (PVDG) Model l Data suppliers publish data to the Grid l Users request raw or derived data from Grid, without needing to know u Where data is located u Whether data is stored or computed l User can easily determine u What it will cost to obtain data u Quality of derived data l PVDG serves requests efficiently, subject to global and local policy constraints

14. ARGONNE  CHICAGO Ian Foster PVDG Scenario User requests may be satisfied via a combination of data access and computation at local, regional, and central sites

15. ARGONNE  CHICAGO Ian Foster User View of PVDG Architecture

16. ARGONNE  CHICAGO Ian Foster Other Activities Relevant to Data Grids l Simulation activities u MONARC, MicroGrid l Globus Data Grid/replica mgmt services u GridFTP: secure high-performance FTP u Replica catalog/replica management l Grid Data Management Pilot (GDMP) u Being used to move data CERN->Caltech u Uses GridFTP u http://cmsdoc.cern.ch/cms/grid/

17. ARGONNE  CHICAGO Ian Foster Example Tech Developments: Globus Data Grid Services Library Program Legend globus-url-copy Replica Programs Custom Servers globus_gass_copy globus_ftp_client globus_ftp_control globus_commonGSI (security) globus_ioOpenLDAP client globus_replica_catalog globus_replica_manager Custom Clients globus_gass_transfer globus_gass Already exist

18. ARGONNE  CHICAGO Ian Foster Example Technology Developments: Quality of Service for Bulk Transfer When a reservation begins, the bulk-transfer backs off When a reservation ends, the bulk-transfer speeds up The competitive UDP traffic never interferes GARA: www.mcs.anl.gov/qos

19. ARGONNE  CHICAGO Ian Foster Summary l New data-intensive applications require a new type of infrastructure: “Data Grids” l Concerns and infrastructure requirements have much in common with other “Grids” l Development requires substantial R&D in caching, security, policy, QoS, etc., etc. l Existing technology base enables contruction of Data Grids to start now www.globus.org www.griphyn.org www.gridforum.org www.ppdg.net grid.web.cern.ch