1. Grid Computing: Concepts, Applications, and Technologies
Dheeraj Bhardwaj
Department of Computer Science and Engineering
Indian Institute of Technology, Delhi

2. Outline The technology landscape Grid computing The Globus Toolkit
Applications and technologies
Data-intensive; distributed computing; collaborative; remote access to facilities
Grid infrastructure
Open Grid Services Architecture
Global Grid Forum
Summary and conclusions

3. Outline The technology landscape Grid computing The Globus Toolkit
Applications and technologies
Data-intensive; distributed computing; collaborative; remote access to facilities
Grid infrastructure
Open Grid Services Architecture
Global Grid Forum
Summary and conclusions

4. Living in an Exponential World (1) Computing & Sensors
Moore’s Law: transistor count doubles each 18 months
Magnetohydro-
dynamics
star formation

5. Living in an Exponential World: (2) Storage
Storage density doubles every 12 months
Dramatic growth in online data (1 petabyte = 1000 terabyte = 1,000,000 gigabyte)
2000 ~0.5 petabyte
2005 ~10 petabytes
2010 ~100 petabytes
2015 ~1000 petabytes?
Transforming entire disciplines in physical and, increasingly, biological sciences; humanities next?

6. Data Intensive Physical Sciences
High energy & nuclear physics
Including new experiments at CERN
Gravity wave searches
LIGO, GEO, VIRGO
Time-dependent 3-D systems (simulation, data)
Earth Observation, climate modeling
Geophysics, earthquake modeling
Fluids, aerodynamic design
Pollutant dispersal scenarios
Astronomy: Digital sky surveys

7. Ongoing Astronomical Mega-Surveys
Large number of new surveys
Multi-TB in size, 100M objects or larger
In databases
Individual archives planned and under way
Multi-wavelength view of the sky
> 13 wavelength coverage within 5 years
Impressive early discoveries
Finding exotic objects by unusual colors
L,T dwarfs, high redshift quasars
Finding objects by time variability
Gravitational micro-lensing
MACHO
2MASS
SDSS
DPOSS
GSC-II
COBE MAP
NVSS
FIRST
GALEX
ROSAT
OGLE
...

8. Coming Floods of Astronomy Data
The planned Large Synoptic Survey Telescope will produce over 10 petabytes per year by 2008!
All-sky survey every few days, so will have fine-grain time series for the first time

9. Data Intensive Biology and Medicine
Medical data
X-Ray, mammography data, etc. (many petabytes)
Digitizing patient records (ditto)
X-ray crystallography
Molecular genomics and related disciplines
Human Genome, other genome databases
Proteomics (protein structure, activities, …)
Protein interactions, drug delivery
Virtual Population Laboratory (proposed)
Simulate likely spread of disease outbreaks
Brain scans (3-D, time dependent)

10. A Brain is a Lot of Data! (Mark Ellisman, UCSD)
And comparisons must be
made among many
We need to get to one micron to know location of every cell. We’re just now
starting to get to 10 microns – Grids will help get us there and further

11. An Exponential World: (3) Networks (Or, Coefficients Matter …)
Network vs. computer performance
Computer speed doubles every 18 months
Network speed doubles every 9 months
Difference = order of magnitude per 5 years
1986 to 2000
Computers: x 500
Networks: x 340,000
2001 to 2010
Computers: x 60
Networks: x 4000
Moore’s Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan-2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.

12. Outline The technology landscape Grid computing The Globus Toolkit
Applications and technologies
Data-intensive; distributed computing; collaborative; remote access to facilities
Grid infrastructure
Open Grid Services Architecture
Global Grid Forum
Summary and conclusions

13. Evolution of the Scientific Process
Pre-electronic
Theorize &/or experiment, alone or in small teams; publish paper
Post-electronic
Construct and mine very large databases of observational or simulation data
Develop computer simulations & analyses
Exchange information quasi-instantaneously within large, distributed, multidisciplinary teams

14. Evolution of Business Pre-Internet Post-Internet
Central corporate data processing facility
Business processes not compute-oriented
Post-Internet
Enterprise computing is highly distributed, heterogeneous, inter-enterprise (B2B)
Outsourcing becomes feasible => service providers of various sorts
Business processes increasingly computing- and data-rich

15. The Grid
“Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”

16. A Comparison SERIAL Fetch/Store Compute PARALLEL Fetch/Store
Compute/ communicate
Cooperative game
GRID
Fetch/Store
Discovery of Resources
Interaction with remote application
Authentication / Authorization
Security
Compute/Communicate
Etc

17. A Comparison SERIAL Fetch/Store Compute PARALLEL Fetch/Store
Compute/ communicate
Cooperative game
GRID
Fetch/Store
Discovery of Resources
Interaction with remote application
Authentication / Authorization
Security
Compute/Communicate
Etc

18. Distributed Computing vs. GRID
Grid is an evolution of distributed computing
Dynamic
Geographically independent
Built around standards
Internet backbone
Distributed computing is an “older term”
Typically built around proprietary software and network
Tightly couples systems/organization

19. Web vs. GRID
Web
Uniform naming access to documents
Grid - Uniform, high performance access to computational resources
Software Catalogs
Sensor nets
Colleges/R&D Labs

20. Is the World Wide Web a Grid ?
Seamless naming? Yes
Uniform security and Authentication? No
Information Service? Yes or No
Co-Scheduling? No
Accounting & Authorization ? No
User Services? No
Event Services? No
Is the Browser a Global Shell ? No

21. What does the World Wide Web bring to the Grid ?
Uniform Naming
A seamless, scalable information service
A powerful new meta-data language: XML
XML will be standard language for describing information in the grid
SOAP – simple object access protocol
Uses XML for encoding. HTML for protocol
SOAP may become a standard RPC mechanism for Grid services
Portal Ideas

22. The Ultimate Goal
In future I will not know or care where my application will be executed as I will acquire and pay to use these resources as I need them

23. Why Grids?
Large-scale science and engineering are done through the interaction of people, heterogeneous computing resources, information systems, and instruments, all of which are geographically and organizationally dispersed.
The overall motivation for “Grids” is to facilitate the routine interactions of these resources in order to support large-scale science and Engineering.

24. An Example Virtual Organization: CERN’s Large Hadron Collider
1800 Physicists, 150 Institutes, 32 Countries
100 PB of data by 2010; 50,000 CPUs?

25. Grid Communities & Applications: Data Grids for High Energy Physics
Tier2 Centre ~1 TIPS
Online System
Offline Processor Farm
~20 TIPS
CERN Computer Centre
FermiLab ~4 TIPS
France Regional Centre
Italy Regional Centre
Germany Regional Centre
Institute
Institute ~0.25TIPS
Pentium II 300 MHz
Physicist workstations
~100 MBytes/sec
~622 Mbits/sec
~1 MBytes/sec
HPSS
There is a “bunch crossing” every 25 nsecs.
There are 100 “triggers” per second
Each triggered event is ~1 MByte in size
Physicists work on analysis “channels”.
Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server
Physics data cache
~PBytes/sec
~622 Mbits/sec or Air Freight (deprecated)
Caltech ~1 TIPS
Tier 0
Tier 1
Tier 2
Tier 4
1 TIPS is approximately 25,000
SpecInt95 equivalents

26. Intelligent Infrastructure: Distributed Servers and Services

27. The Grid Opportunity: eScience and eBusiness
Physicists worldwide pool resources for peta-op analyses of petabytes of data
Civil engineers collaborate to design, execute, & analyze shake table experiments
An insurance company mines data from partner hospitals for fraud detection
An application service provider offloads excess load to a compute cycle provider
An enterprise configures internal & external resources to support eBusiness workload

28. The Grid: A Brief History
Early 90s
Gigabit testbeds, metacomputing
Mid to late 90s
Early experiments (e.g., I-WAY), academic software projects (e.g., Globus, Legion), application experiments
2002
Dozens of application communities & projects
Major infrastructure deployments
Significant technology base (esp. Globus ToolkitTM)
Growing industrial interest
Global Grid Forum: ~500 people, 20+ countries

29. Challenging Technical Requirements
Dynamic formation and management of virtual organizations
Online negotiation of access to services: who, what, why, when, how
Establishment of applications and systems able to deliver multiple qualities of service
Autonomic management of infrastructure elements
Open Grid Services Architecture

30. Grid Concept (Take 1) Analogy with the electrical power grid
“On-demand” access to ubiquitous distributed computing
Transparent access to multi-petabyte distributed data bases
Easy to plug resources into
Complexity of the infrastructure is hidden
“When the network is as fast as the computer's internal links, the machine disintegrates across the net into a set of special purpose appliances” (George Gilder)

31. Grid Vision (Take 2)
e-Science and information utilities Science increasingly done through distributed global collaborations between people, enabled by the Internet
Using very large data collections, terascale computing resources, and high performance visualisation
Derived from instruments and facilities controlled and shared via the infrastructure
Scaling x1000 in processing power, data, bandwidth

32. Elements of the Problem
Resource sharing
Computers, storage, sensors, networks, …
Heterogeneity of device, mechanism, policy
Sharing conditional: negotiation, payment, …
Coordinated problem solving
Integration of distributed resources
Compound quality of service requirements
Dynamic, multi-institutional virtual orgs
Dynamic overlays on classic org structures
Map to underlying control mechanisms

33. The Grid World: Current Status
Dozens of major Grid projects in scientific & technical computing/research & education
Considerable consensus on key concepts and technologies
Open source Globus Toolkit™ a de facto standard for major protocols & services
Industrial interest emerging rapidly
IBM, Platform, Microsoft, Sun, Compaq, …
Opportunity: convergence of eScience and eBusiness requirements & technologies

34. Outline The technology landscape Grid computing The Globus Toolkit
Applications and technologies
Data-intensive; distributed computing; collaborative; remote access to facilities
Grid infrastructure
Open Grid Services Architecture
Global Grid Forum
Summary and conclusions

35. Grid Technologies: Resource Sharing Mechanisms That …
Address security and policy concerns of resource owners and users
Are flexible enough to deal with many resource types and sharing modalities
Scale to large number of resources, many participants, many program components
Operate efficiently when dealing with large amounts of data & computation

36. Aspects of the Problem
Need for interoperability when different groups want to share resources
Diverse components, policies, mechanisms
E.g., standard notions of identity, means of communication, resource descriptions
Need for shared infrastructure services to avoid repeated development, installation
E.g., one port/service/protocol for remote access to computing, not one per tool/appln
E.g., Certificate Authorities: expensive to run
A common need for protocols & services

37. The Hourglass Model Focus on architecture issues Design principles
Propose set of core services as basic infrastructure
Use to construct high-level, domain-specific solutions
Design principles
Keep participation cost low
Enable local control
Support for adaptation
“IP hourglass” model
A p p l i c a t i o n s
Diverse global services
Core
services
Local OS

38. Layered Grid Architecture (By Analogy to Internet Architecture)
Application
Internet
Transport
Application
Link
Internet Protocol Architecture
Collective
“Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services
We define Grid architecture in terms of a layered collection of protocols.
Fabric layer includes the protocols and interfaces that provide access to the resources that are being shared, including computers, storage systems, datasets, programs, and networks. This layer is a logical view rather then a physical view. For example, the view of a cluster with a local resource manager is defined by the local resource manger, and not the cluster hardware. Likewise, the fabric provided by a storage system is defined by the file system that is available on that system, not the raw disk or tapes.
The connectivity layer defines core protocols required for Grid-specific network transactions. This layer includes the IP protocol stack (system level application protocols [e.g. DNS, RSVP, Routing], transport and internet layers), as well as core Grid security protocols for authentication and authorization.
Resource layer defines protocols to initiate and control sharing of (local) resources. Services defined at this level are gatekeeper, GRIS, along with some user oriented application protocols from the Internet protocol suite, such as file-transfer.
Collective layer defines protocols that provide system oriented capabilities that are expected to be wide scale in deployment and generic in function. This includes GIIS, bandwidth brokers, resource brokers,….
Application layer defines protocols and services that are parochial in nature, targeted towards a specific application domain or class of applications. These are are are … arrgh
Resource
“Sharing single resources”: negotiating access, controlling use
Connectivity
“Talking to things”: communication (Internet protocols) & security
Fabric
“Controlling things locally”: Access to, & control of, resources

39. Globus Toolkit™
A software toolkit addressing key technical problems in the development of Grid-enabled tools, services, and applications
Offer a modular set of orthogonal services
Enable incremental development of grid-enabled tools and applications
Implement standard Grid protocols and APIs
Available under liberal open source license
Large community of developers & users
Commercial support

40. General Approach Define Grid protocols & APIs
Protocol-mediated access to remote resources
Integrate and extend existing standards
“On the Grid” = speak “Intergrid” protocols
Develop a reference implementation
Open source Globus Toolkit
Client and server SDKs, services, tools, etc.
Grid-enable wide variety of tools
Globus Toolkit, FTP, SSH, Condor, SRB, MPI, …
Learn through deployment and applications

41. Key Protocols The Globus Toolkit™ centers around four key protocols
Connectivity layer:
Security: Grid Security Infrastructure (GSI)
Resource layer:
Resource Management: Grid Resource Allocation Management (GRAM)
Information Services: Grid Resource Information Protocol (GRIP) and Index Information Protocol (GIIP)
Data Transfer: Grid File Transfer Protocol (GridFTP)
Also key collective layer protocols
Info Services, Replica Management, etc.