Grid Architecture Developments Aslam Parvez Memon & Shakil Akhtar Shaheed Zulifkar Ali Bhutto Institute of Science and Technology (SZABIST), Karachi Pakistan. December 21, 2006

Outlines Defining Grid Computing What is Grid and Grid Computing Grid Technology Why Grid Study of Grid Computing Grid Technology Problem Space Virtual Organization Virtual Organization Problem Space Grid Architecture Some Solutions Globus Toolkit Working With Gird Key Concepts of GT4 Major Grid Projects Research Bodies Global Community Research Areas The Research Processes

Ian Foster’s 3 point checklist A Grid is a system that is able to coordinate “resources that are not subject to centralized control” Use “standard, open, general-purpose protocols and interfaces” “to deliver nontrivial qualities of service.”

Defining Grid Computing There are several competing definitions for “The Grid” and Grid computing These definitions tend to focus on: Implementation of Distributed computing A common set of interfaces, tools and APIs Some stress the inter-institutional aspect of grids and Virtual Organizations “The Virtualization of Resources” abstraction of resources

What is Grid and Grid Computing? Grid computing must provide basic functions 1.resource discovery and information collection & publishing 2.data management on and between resources 3.process management on and between resources 4.common security mechanism underlying the above 5.process and session recording/accounting

Grid Technology Emerging enabling technology. Natural evolution of distributed systems and the Internet. Middleware supporting network of systems to facilitate sharing, standardization and openness. Infrastructure and application model dealing with sharing of compute cycles, data, storage and other resources. Publicized by prominent industries as on-demand computing, utility computing, etc. Move towards delivering “computing” to masses similar to other utilities (electricity and voice communication). Currently used for high performance computing however the trend is towards Service Oriented Applications (SOA).

Why Grid? What can the grid do that existing technology cannot do? Grid infrastructure and application architecture form a global computing framework facilitating sharing of resources and schedulability of jobs by matching their needs with available pool of compute and storage resources. Compute cycles can be tapped on demand from sources other then yours. Wasted cycles from idle sources can be utilized for use in needed application. Grid is molding computing into an utility similar to utilities we are used to: electricity and telephone.

Study of Grid Computing Components: Core, system defined and user defined Infrastructure Application model Standards Application Programming Interfaces Technology Support (enabling technologies) Job submission and associated functions Service creation and deployment and related functions

Grid Technology Problem Space Grid technologies and infrastructures support the sharing and coordinated use of diverse resources in dynamic, distributed “virtual organizations”. Grid technologies are distinct from technology trends such as Internet, enterprise, distributed and peer-to- peer computing. But these technologies can benefit from growing into the “problem space” addressed by grid technologies.

The Grid Problem Flexible, secure, coordinated sharing of computation among dynamic collections of individuals, institutions, and resources Enable communities (“virtual organizations”) to share geographically distributed resources as they pursue common goals -- assuming the absence of… central location central control omniscience existing trust relationships

Virtual Organization Grid’s virtual organizations (VOs) concept provides seamless access to federated heterogeneous resources—computers, mobile devices, network bandwidth, storage, databases, scientific instruments, servers etc. by creating illusion of supercomputing infrastructure. A grid user can have on demand access to such resources, distributed across various organizations in different geographical locations, yet in a controlled and secure resource sharing environment.

Elements of the Problem Resource sharing Computers, storage, sensors, networks, … Sharing always conditional: issues of trust, policy, negotiation, payment, … Coordinated problem solving Beyond client-server: distributed data analysis, computation, collaboration, … Dynamic, multi-institutional virtual orgs Community overlays on classic org structures Large or small, static or dynamic

The Programming Problem Applications require resources (compute power, storage, data, instruments, displays) at many sites for many users. Some requirements: Abstractions and models to increase speed/robustness/etc. of development Tools to ease application development and diagnose common problems, ease deployment Code/tool sharing to allow reuse of code components developed by others

Grid must suspport computational workflows Locate “suitable” computers Authenticate with appropriate sites Allocate resources on those computers Initiate computation on those computers Configure those computations Select “appropriate” communication methods Compute with “suitable” algorithms Access data files, return output Respond “appropriately” to resource changes

Grid Requirements identity & authentication authorization & policy resource/service discovery resource allocation (co-)reservation, workflow remote data access rapid data transfer monitoring intrusion detection resource management accounting fault management system evolution and more…

Grid Computing - Functions Grid computing must provide typically these basic functions (Foster/Kesselman) resource discovery and information collection & publishing data management on and between resources process management on and between resources common security mechanism underlying the above In addition, it should include: process and session recording/accounting

Grid Architecture Architecture identifies the fundamental system components, specifies purpose and function of these components, and indicates how these components interact with each other. Grid architecture is a protocol architecture, with protocols defining the basic mechanisms by which VO users and resources negotiate, establish, manage and exploit sharing relationships. Grid architecture is also a services standards-based open architecture that facilitates extensibility, interoperability, portability and code sharing. API and Toolkits are also being developed.

Layered Grid Architecture Fabric Layer - provides the local services of a resource: computational, storage, network Connective Layer - core communication and authentication protocols Enables exchange of data between fabric layer resources Security and authentication important here

Layered Grid Architecture (cont.) Resource Layer – enables resource sharing Builds on connectivity layer to control and access resources (Ex: data servers) Collective Layer - coordinates interactions across multiple resources Ties multiple resources and services together (Ex: metacatalogues) Application Layer - user applications use collective, resource, and connective layers to perform grid operations in a virtual organization

Some Solutions Middleware Toolkits: not all speak (or spoke) Globus: Condor Globus Toolkit Legion/Avaki Condor (now Sun Grid Engine) Unicore Higher Level Toolkits (build on Globus) JavaCoG GridPortal Toolkit, Grid Portal Development Toolkit (GPDK) Condor-G SGE

The Globus Toolkit Open-source reference software base for developing Grid infrastructure and applications Implements GGF standards Service-oriented Services can be decoupled from any fixed resource A service consumes resources, but how is not most important A better base abstraction for managing dependability, end-to-end quality of service Slide Courtesy of Ian Foster presentation at Comdex04

Globus Protocols - Connectivity Layer Grid Security Infrastructure (GSI): Authentication/authorization, message protection across institutions Single sign-on, delegation, identity mapping Public key technology Certificate authorities, certificate & key management Ian Foster, et. al., “Anatomy of the Grid”

Globus Protocols - Collective Layer Metadirectory services Resource brokers Condor Co-reservation/co-allocation services Workflow management services Ian Foster, et. al., “Anatomy of the Grid”

Globus Protocols – Resource Layer Grid Resource Allocation Management (GRAM) Remote allocation, control of compute resources Furnishes information on state of the resources to the Metacomputing Directory Service (MDS) GridFTP High-performance data access and transport Grid Resource Information Service (GRIS) Access to structure and state info (MDS) All built on connectivity layer Ian Foster, et. al., “Anatomy of the Grid”

Grid Security Infrastructure (GSI) Public key cryptography Encryption relies on two keys, related mathematically so that if either key encrypts a message, the other must be used to decrypt it One key is public, the other is kept private A user proves own identity by encrypting a message; if the public key can decrypt, the user is indeed holding the private key No password is ever exchanged Ian Foster, et. al., “Anatomy of the Grid”

Working With Grids 1.A user enrolls himself or his machine with grid system 2.The user establishes his identity with CA (this process may require alternate ways other than internet) 3.The CA takes steps to make sure that the user is in fact who, he claims to be. 4.The CA makes special certificate available to the software, which needs to check the identity of user and his requests to the grid system. 5.Steps 1-4 may be repeated for the donor machine(s). A user must keep his security credentials secure. 6.User installs the software provided by the grid system to use the grid and/ donate the machine. The software may be auto configured or manual by the user, this configuration is required for: Grid nodes management Machine identification information Implement constraints on resources’ access such as time, type etc. Providing user’s IDs on other machines that exist on grid.

1.A user is required to login to a grid system using user ID that is enrolled in the grid. 2.The user can use gird system IDs or operating system IDs, which ever is enrolled with grid, but grid system ID is recommended for two reasons: It eliminates the need for matching IDs form machine to machine A user can access entire grid as a one large virtual computer using common ID across the grid. 3.Globus, as mentioned above uses proxy login model, which keeps a user logged in for a specified amount of time, even if a user logs off and logs back on the operating system, and even if the machine is rebooted. 4.Once the user is logged in he can query the grid or submit job using localization interfaces.

Key Concepts for GT4 OGSA, WSRF, and GT4 These are basic architecture components for GT4 Open Grid Services Architecture (OGSA) Web Services: OGSA, WSRF, and GT4 are based on standard Web Services technologies such as SOAP and WSDL. Ned to be familiar with the Web Services architecture and languages. The Web Services Resource Framework: WSRF is the core of GT4. Based on WS-Resources and Web Services, and grid computing Java & XML: to use GT4, you need to be able to program in Java, and to understand basic XML.

OGSA Key Requirements Interoperability and Support for Dynamic and Heterogeneous Environments Resource Sharing Across Organizations Job Execution Data Services Security Optimization Quality of Service (QoS) Assurance Administrative Cost Reduction Scalability Availability Ease of Use and Extensibility

OGSA Defines Basic Capabilities Infrastructure Services Execution Management Services Data Services Resource Management Services Security Services Information Services Security Considerations

GT Architecture GT4 comprises both a set of service implementations (“server” code) and associated “client” libraries. GT4 provides both Web services (WS) components and non-WS components All GT4 WS components use WS-Interoperability- compliant transport and security mechanisms can interoperate with each other and with other WS components. All GT4 components support X.509 certificates both WS and non-WS client can use the same credentials to authenticate with any GT4 WS or non-WS component.

GT4 Services Nine GT4 services implement Web services (WS) interfaces: Job management (GRAM) Reliable File Transfer Delegation Monitoring and Discovery System (MDS) MDS-Index, MDS-Trigger, and MDSArchive Community Authorization (CAS) OGSA-DAI data access and integration Grid TeleControl Protocol (GTCP) Grid remote instrumentation control

OGSA, WSRF, GT4 Relationship Diagram

WS Software stack used by GT4 WSRF HTTP Server Apache HTTP Server Application Server Apache Tomcat SOAP Engine Apache AXIS Supports wsdl2java tool - build Java proxies and skeletons from WSDL docs. Web Service User App

GT4 Roadmap

Major Grid Projects Earth System Grid, Virtual Observatory, European Data Grid, GriPhyN Project, PPDG, HEPGRID, Virtual Laboratory Grid, NEESGRID, GEOSIDE, Fusion Grid, IPG Grid, ActiveSeets China National Grid (CNGrid), China Science Grid (CSGrid) categoryid=84http:// categoryid=84 China Semantic Grid, Shanghai City Information Grid

Research Bodies and Consortiums Global Grid Forum (GGF), OASIS, DMTF, CIM, WBEM, W3C, Globus Alliance, GridBus Project, Condor Project Home Page, Legion Project, Unicore,

Global Community Slide Courtesy of Ian Foster

Applications Proof concepts Academic prototype applications Long terms applications Grid based Agro-MIS Grid based LRMIS Grid Based LLDP And more……..

Research Areas Protocols and Standards Development Grid Security Service Oriented Architecture Resource Management Scheduling Grid Operating Environments Grid Software Development Environments Quality of Services Grid Localization Grid Simulations Toolkits and Portals developmentb

The Process Literature Review Research Question/ Hypothesis Analysis (Generic Specifications) Design (Technical Specification) Implementation Testing Documentation Presentation

Thank You!