1. CS6703 GRID AND CLOUD COMPUTING Unit 2

2. UNIT II GRID SERVICES
Introduction to Open Grid Services Architecture (OGSA) – Motivation – Functionality Requirements – Practical & Detailed view of OGSA/OGSI – Data intensive grid service models – OGSA services.

3. Grid Architecture

4. The Hourglass Model Focus on architecture issues
Propose set of core services as basic infrastructure
Used to construct high-level, domain-specific solutions (diverse)
Design principles
Keep participation cost low
Enable local control
Support for adaptation
“IP hourglass” model
A p p l i c a t i o n s
Diverse global services
Core
services
Local OS

5. Layered Grid Architecture (By Analogy to Internet Architecture)
Application
Internet
Transport
Application
Link
Internet Protocol Architecture
Collective
“Coordinating multiple resources”: ubiquitous infrastructure services, app-specific distributed services
We define Grid architecture in terms of a layered collection of protocols.
Fabric layer includes the protocols and interfaces that provide access to the resources that are being shared, including computers, storage systems, datasets, programs, and networks. This layer is a logical view rather then a physical view. For example, the view of a cluster with a local resource manager is defined by the local resource manger, and not the cluster hardware. Likewise, the fabric provided by a storage system is defined by the file system that is available on that system, not the raw disk or tapes.
The connectivity layer defines core protocols required for Grid-specific network transactions. This layer includes the IP protocol stack (system level application protocols [e.g. DNS, RSVP, Routing], transport and internet layers), as well as core Grid security protocols for authentication and authorization.
Resource layer defines protocols to initiate and control sharing of (local) resources. Services defined at this level are gatekeeper, GRIS, along with some user oriented application protocols from the Internet protocol suite, such as file-transfer.
Collective layer defines protocols that provide system oriented capabilities that are expected to be wide scale in deployment and generic in function. This includes GIIS, bandwidth brokers, resource brokers,….
Application layer defines protocols and services that are parochial in nature, targeted towards a specific application domain or class of applications. These are are are … arrgh
Resource
“Sharing single resources”: negotiating access, controlling use
Connectivity
“Talking to things”: communication (Internet protocols) & security
Fabric
“Controlling things locally”: Access to, & control of, resources

6. We define Grid architecture in terms of a layered collection of protocols.
Fabric layer includes the protocols and interfaces that provide access to the resources that are being shared, including computers, storage systems, datasets, programs, and networks. This layer is a logical view rather then a physical view. For example, the view of a cluster with a local resource manager is defined by the local resource manger, and not the cluster hardware. Likewise, the fabric provided by a storage system is defined by the file system that is available on that system, not the raw disk or tapes.
The connectivity layer defines core protocols required for Grid-specific network transactions. This layer includes the IP protocol stack (system level application protocols [e.g. DNS, RSVP, Routing], transport and internet layers), as well as core Grid security protocols for authentication and authorization.
Resource layer defines protocols to initiate and control sharing of (local) resources. Services defined at this level are gatekeeper, GRIS, along with some user oriented application protocols from the Internet protocol suite, such as file-transfer. (Grid Resource Information Service is the repository of local resource information derived from information providers)
Collective layer defines protocols that provide system oriented capabilities that are expected to be wide scale in deployment and generic in function. This includes GIIS, bandwidth brokers, resource brokers,(Grid Index Information Service: (GIIS): represents a centralized MDS server that provides information about all of your resources) *Master Data Services (MDS) enables your organization to manage a trusted version of data
Application layer defines protocols and services that are parochial in nature, targeted towards a specific application domain or class of applications.

7. Example: Data Grid Architecture
App
Discipline-Specific Data Grid Application
Collective
(App)
Coherency control, replica selection, task management, virtual data catalog, virtual data code catalog, …
Replica catalog, replica management, co-allocation, certificate authorities, metadata catalogs,
Collective
(Generic)
Access to data, access to computers, access to network performance data, …
Resource
Communication, service discovery (DNS), authentication, authorization, delegation
Connect
Fabric
Storage systems, clusters, networks, network caches, …

8. Simulation tools GridSim – job scheduling
SimGrid – single client multiserver scheduling
Bricks – scheduling
GangSim- Ganglia VO
OptoSim – Data Grid Simulations
G3S – Grid Security services Simulator – security services

9. Simulation tool
GridSim is a Java-based toolkit for modeling, and simulation of distributed resource management and scheduling for conventional Grid environment.
GridSim is based on SimJava, a general purpose discrete-event simulation package implemented in Java.
All components in GridSim communicate with each other through message passing operations defined by SimJava.

10. Salient features of the GridSim
It allows modeling of heterogeneous types of resources.
Resources can be modeled operating under space- or time-shared mode.
Resource capability can be defined (in the form of MIPS (Million Instructions Per Second) benchmark.
Resources can be located in any time zone.
Weekends and holidays can be mapped depending on resource’s local time to model non-Grid (local) workload.
Resources can be booked for advance reservation.
Applications with different parallel application models can be simulated.

11. Salient features of the GridSim
Application tasks can be heterogeneous and they can be CPU or I/O intensive.
There is no limit on the number of application jobs that can be submitted to a resource.
Multiple user entities can submit tasks for execution simultaneously in the same resource, which may be time-shared or space-shared. This feature helps in building schedulers that can use different market-driven economic models for selecting services competitively.
Network speed between resources can be specified.
It supports simulation of both static and dynamic schedulers.
Statistics of all or selected operations can be recorded and they can be analyzed using GridSim statistics analysis methods.

12. A Modular Architecture for GridSim Platform and Components.
Appn Conf
Res Conf
User Req
Grid Sc
Output
Application, User, Grid Scenario’s input and Results
Grid Resource Brokers or Schedulers …
Appn modeling
Res entity
Info serv
Job mgmt
Res alloc
Statis
GridSim Toolkit
Single CPU
SMPs
Clusters
Load
Netw
Reservation
Resource Modeling and Simulation
SimJava
Distributed SimJava
Basic Discrete Event Simulation Infrastructure
PCs
Workstation
Distributed Resources
Virtual Machine

13. What is the OGSA Standard?
Acronym for Open Grid Service Architecture
OGSA define how different components in grid interact
Open Grid Services Architecture (OGSA) is a set of standards defining the way in which information is shared among diverse components of large, heterogeneous grid systems.
In this context, a grid system is a scalable wide area network (WAN) that supports resource sharing and distribution.

14. major goals of OSGA
Identify the use cases that can drive the OGSA platform components.
Identify and define the core OGSA platform components.
Define hosting and platform specific bindings.
Define resource models and resource profiles with interoperable solutions.

15. Functional requirements of OGSA.
Interoperability and Support for Dynamic and Heterogeneous Environments
Resource Sharing Across Organizations
Optimization
Quality of Service (QoS) Assurance
Job Execution
Data Services
Security
Administrative Cost Reduction
Scalability
Availability
Ease of Use and Extensibility

16. Architecture of OGSA Comprised of 4 main layers
Physical and Logical Resources Layer
Web Service Layer
OGSA Architected Grid Services Layer
Grid Applications Layer

17. OGSA Architecture

18. OGSA Architecture - Physical and Logical Resources Layer
Physical resources are: servers, storage, network
Logical resources manage physical resources
Examples of logical resources: database managers, workflow managers
MIE456

19. OGSA Architecture - Web Services Layer
Web service is software available online that could interact with other software using XML
Consists of Open Grid Services Infrastructure (OGSI) sub-layer which specifies grid services and provide consistent way to interact with grid services
Also extends Web Service Capabilities
Consists of 5 interfaces:
Factory: provide way for creation of new grid services
Life Cycle: Manages grid service life cycles
State Management: Manage grid service states
Service Groups: collection of indexed grid services
Notification: Manages notification between services & resources

20. OGSA Architecture - Web Services Layer (OGSI)

21. OGSA Architecture – OGSA Architected Services - Layer
Classified into 3 service categories
Grid Core Services
Grid Program Execution Services
Grid Data Services

22. OGSA Architected Services – Grid Core Services
Composed of 4 main types of services:
Service Management: assist in installation, maintenance, & troubleshooting tasks in grid system
Service Communication: include functions that allow grid services to communicate
Policy Services: Provide framework for creation, administration & management of policies for system operation
Security Services: provide authentication & authorization mechanisms to ensure systems interoperate securely

23. OGSA Architected Services – Grid Program Execution Services
Supports unique grid systems in high performance computing, collaboration, parallelism
Support virtualization of resource processing

24. OGSA Architected Services – Grid Data Services
Support data virtualization
Provide mechanism for access to distributed resources such as databases, files

25. OGSA Architecture – OGSA Architected Services - Layer

26. OGSA Architecture – Grid Applications Layer
This layer comprise of applications that use the grid architected services

27. Functional requirements of OGSA
Interoperability and Support for Dynamic and Heterogeneous Environments
Resource Sharing Across Organizations
Optimization
Quality of Service (QoS) Assurance
Job Execution
Data Services
Security
Administrative Cost Reduction
Scalability
Availability
Ease of Use and Extensibility

28. Interoperability and Support for Dynamic and Heterogeneous Environments
The need to support heterogeneous systems leads to requirements that include the following:
• Resource virtualization. Essential to reduce the complexity of managing heterogeneous systems and to handle diverse resources in a unified way.
• Common management capabilities. Simplifying administration of a heterogeneous system requires mechanisms for uniform and consistent management of resources. A minimum set of common manageability capabilities is required.
• Resource discovery and query. Mechanisms are required for discovering resources with desired attributes and for retrieving their properties. Discovery and query should handle a highly dynamic and heterogeneous system.
• Standard protocols and schemas. Important for interoperability. In addition, standard protocols are also particularly important as their use can simplify the transition to using Grids.

29. Resource Sharing Across Organizations
One major purpose of OGSA is to support resource sharing and utilization across administrative domains, whether different work units within an enterprise or even different institutions. Resource sharing requirements include: • Global name space. To ease data and resource access. OGSA entities should be able to access other OGSA entities transparently, subject to security constraints, without regard to location or replication. • Metadata services. Important for finding, invoking, and tracking entities. It should be possible to allow for access to and propagation, aggregation, and management of entity metadata across administrative domains. • Site autonomy. Mechanisms are required for accessing resources across sites while respecting local control and policy • Resource usage data. Mechanisms and standard schemas for collecting and exchanging resource usage (i.e., consumption) data across organizations—for the purpose of accounting, billing, etc.

30. Optimization Quality of Service (QoS) Assurance
Optimization refers to techniques used to allocate resources effectively to meet consumer and supplier requirements. Optimization applies to both suppliers (supply-side) and consumers (consume-side) of resources and services
Optimization
Quality of Service (QoS) Assurance
Services such as job execution and data services must provide the agreed-upon QoS. Key QoS dimensions include, but are not limited to, availability, security, and performance.
QoS assurance requirements include:
• Service level agreement. QoS should be represented by agreements which are established through negotiation between service requester and provider prior to service execution. Standard mechanisms should be provided to create and manage agreements.
• Service level attainment. If the agreement requires attainment of Service Level, the resources used by the service should be adjusted so that the required QoS is maintained. Therefore, mechanisms for monitoring service quality, estimating resource utilization, and planning for and adjusting resource usage are required.
• Migration. It should be possible to migrate executing services or applications to adjust workloads for performance or availability

31. Job Execution
Functions such as scheduling, provisioning, job control and exception handling of jobs must be supported, even when the job is distributed over a great number of heterogeneous resources.
Job execution requirements include:
• Support for various job types. Execution of various types of jobs must be supported including simple jobs and complex jobs such as workflow and composite services.
• Job management. It is essential to be able to manage jobs during their entire lifetimes, types of groupings of jobs (e.g., workflows, job arrays). Mechanisms are also required for controlling the execution of individual job steps as well as orchestration or choreography services.
• Scheduling. The ability to schedule and execute jobs based on such information as specified priority and current allocation of resources is required. It is also required to realize mechanisms for scheduling across administrative domains, using multiple schedulers.
• Resource provisioning. To automate the complicated process of resource allocation, deployment, and configuration. It must be possible to deploy the required applications and data to resources and configure them automatically.

32. Data Services
require support for the sharing and integration of distributed data, for example enabling access to information stored in databases that are managed and administered independently, with appropriate security assurances.
Data services requirements include:
• Policy specification & management. Examples include specification of who can access data, where data will be required, what transformations are permitted on the data, whether use is exclusive, what performance or availability is required, how much resources can be used, what consistency is mandated between replicas, and similar constraints.
Data storage. Disk and tape systems, amongst others, store data. Common interfaces support the provision of storage and management of quotas, lifetime, and properties such as encryption and persistency.

33. • Data access. Clients require easy and efficient access to various types of data (such as databases, files, streams and integrated/federated data) through a uniform set of interfaces is required, independent of its physical location or platform, by abstracting underlying data resources. • Data transfer. High-bandwidth transfer of data is required, independent of the physical attributes of the data sources and sinks, which can exploit relevant features of those sources and sinks if required. • Data location management. These services manage where data is physically located, OGSA should support multiple methods for making data available to a client at a given location, according to the policy requirements of the client and of the data resource. Methods should include transfer, copying, caching, and replication of data.

34. Data update. Although some data resources are read only, many if not most provide some users with update privileges. OGSA must provide update facilities which ensure that the specified consistency can be maintained when cached or replicated data is modified.
• Data persistency. Data should be preserved according to specified policy and its association with its metadata should be maintained in accordance with that policy. It should be possible to use one of many possible persistency models.
• Data federation. OGSA should support data integration for heterogeneous and distributed data. Heterogeneous data includes data organized according to different schemas and data stored using different technologies (e.g., relational vs. flat file).

35. Security Safe administration requires controlling access to services through robust security protocols and according to provided security policy.
Security requirements include:
• Authentication and authorization.
• Multiple security infrastructures. Distributed operation implies a need to integrate and interoperate with multiple security infrastructures. OGSA needs to integrate and interoperate with existing security architectures and models.
• Perimeter security solutions. Resources may have to be accessed across organizational boundaries, without compromising local security mechanisms, such as firewall policy and intrusion detection policy.

36. • Isolation. Various kinds of isolation must be ensured, such as isolation of users, performance isolation, and isolation between content offerings within the same Grid system.
• Delegation. Mechanisms that allow for delegation of access rights from service requestors to service providers are required. The risk of misuse of delegated rights must be minimized
• Security policy exchange. Service requestors and providers should be able to exchange dynamically security policy information to establish a negotiated security context between them.
• Intrusion detection, protection, and secure logging. Strong monitoring is required for
intrusion detection and identification of misuses, malicious or otherwise, including virus or worm attacks.

37. Administrative Cost Reduction
The complexity of administering large-scale distributed, heterogeneous systems increases administration costs and the risk of human errors
Policy-based management is required to automate Grid system control, so that its operations conform to the goals of the organization that operates and utilizes the Grid system.
Application contents management mechanisms can facilitate the deployment, configuration, and maintenance of complex systems, by allowing all application-related information to be specified and managed as a single logical unit.
Problem determination mechanisms are needed, so that administrators can recognize and cope quickly with emerging problems.

38. Scalability:A large-scale Grid system can create added value such as drastically reducing job turn around (or elapsed) time, allowing for utilizing huge number of resources, thereby enabling new services.
Availability:
mean-time-to-repair (MTTR) -- heterogeneity of the Grid
Disaster recovery mechanisms are needed so that the operation of a Grid system can be recovered quickly and efficiently in case of natural or human-caused disaster, avoiding long-term service disruption. Remote backup and simplifying or automating recovery procedures is required.
Fault management mechanisms can be required so that running jobs are not lost because of resource faults. Mechanisms are required for monitoring, fault detection, and diagnosis of causes or impacts on running jobs. In addition, automation of fault-handling, using techniques such as checkpoint recovery, is desirable.
Ease of Use and Extensibility: mechanism and policy must be realized via extensible and replaceable components, to permit OGSA to evolve over time and allow users to construct their own mechanisms and policies to meet specific needs.

39. Conclusion
Grid-Computing allows networked resources to be combined and used
Grid-Computing offers great benefit to an organization
OGSA are comprehensive standards which governs grid-computing

40. Open Grid Services Infrastructure (OGSI)
Gives a formal and technical specification of what a grid service is.
Its a excruciatingly(exceedingly elaborate or intense) / incredibly / detailed specification of how Grid Services work.
GT3 includes a complete implementation of OGSI.
It is a formal and technical specification of the concepts described in OGSA.
The Globus Toolkit 3 is an implementation of OGSI.
Some other implementations are OGSI::Lite (Perl)1 and the UNICORE OGSA demonstrator2 from the EU GRIP project.
OGSI specification defines grid services and builds upon web services.

42. The Open Grid Services Infrastructure (OGSI) was published by the Global Grid Forum (GGF) as a proposed recommendation in June 2003.[1] It was intended to provide an infrastructure layer for the Open Grid Services Architecture (OGSA)

46. Open Grid Services Infrastructure (OGSI)
OGSI creates an extension model for WSDL called GWSDL (Grid WSDL). The reason is:
Interface inheritance
Service Data (for expressing state information)
Components:
Lifecycle
State management
Service Groups
Factory
Notification
Handle Map

47. OSGi (Open Service Gateway Initiative) is a Java framework for developing and deploying modular software programs and libraries.
Each bundle is a tightly coupled, dynamically loadable collection of classes, jars, and configuration files that explicitly declare their external dependencies (if any).
OSGi Service Gateway Architecture

48. The framework is conceptually divided into the following areas:
Bundles Bundles are normal jar components with extra manifest headers.
Services The services layer connects bundles in a dynamic way by offering a publish-find- bind model for Plain Old Java Interfaces (POJI) or Plain Old Java Objects (POJO).
Services Registry The application programming interface for management services (ServiceRegistration, ServiceTracker and ServiceReference).
Life-Cycle The application programming interface for life cycle management (install, start, stop, update, and uninstall) for bundles.
Modules The layer that defines encapsulation and declaration of dependencies (how a bundle can import and export code).
Security The layer that handles the security aspects by limiting bundle functionality to pre- defined capabilities.

49. Execution Environment Defines what methods and classes are available in a specific platform. There is no fixed list of execution environments, since it is subject to change as the Java Community Process creates new versions and editions of Java.
However, the following set is currently supported by most OSGi implementations:
CDC-1.0/Foundation-1.0
CDC-1.1/Foundation-1.1
OSGi/Minimum-1.0
OSGi/Minimum-1.1
JRE-1.1
From J2SE-1.2 up to J2SE-1.6

50. Data intensive grid service models
Applications in the grid are normally grouped into two categories
Computation-intensive and Data intensive
Data intensive applications deals with massive amounts of data. The grid system must specially designed to discover, transfer and manipulate the massive data sets.
Transferring the massive data sets is a time consuming task.
Data access method is also known as caching, which is often applied to enhance data efficiency in a grid environment.
By replicating the same data block and scattering them in multiple regions in a grid, users can access the same data with locality of references.

51. Data intensive grid service models
Replication strategies determine when and where to create a replica of the data.
The strategies of replications can be classified into dynamic and static
Static method
The locations and number of replicas are determined in advance and will not be modified.
Replication operation require little overhead
Static strategic cannot adapt to changes in demand, bandwidth and storage variability
Optimization is required to determine the location and number of data replicas.
Dynamic strategies
Dynamic strategies can adjust locations and number of data replicas according to change in conditions
Frequent data moving operations can result in much more overhead the static strategies
Optimization may be determined based on whether the data replica is being created, deleted or moved.
The most common replication include preserving locality, minimizing update costs and maximizing profits .

52. Grid data Access models
In general there are four access models for organizing a data grid as listed here
Monadic method
Hierarchical model
Federation model
Hybrid model

53. Monadic method
This is a centralized data repository model. All data is saved in central data repository.
When users want to access some data they have no submit request directly to the central repository.
No data is replicated for preserving data locality.
For a larger grid this model is not efficient in terms of performance and reliability.
Data replication is permitted in this model only when fault tolerance is demanded.

54. Hierarchical model
It is suitable for building a large data grid which has only one large data access directory
Data may be transferred from the source to a second level center. Then some data in the regional center is transferred to the third level centre.
After being forwarded several times specific data objects are accessed directly by users. Higher level data center has a wider coverage area.
PKI security services are easier to implement in this hierarchical data access model

55. Federation model
It is suited for designing a data grid with multiple source of data supplies.
It is also known as a mesh model
The data is shared the data and items are owned and controlled by their original owners.
Only authenticated users are authorized to request data from any data source.
This mesh model cost the most when the number of grid institutions becomes very large

56. Hybrid model
This model combines the best features of the hierarchical and mesh models.
Traditional data transfer technology such as FTP applies for networks with lower bandwidth.
High bandwidth are exploited by high speed data transfer tools such as GridFTP developed with Globus library.
The cost of hybrid model can be traded off between the two extreme models of hierarchical and mesh-connected grids.

57. Parallel versus Striped Data Transfers
Parallel data transfer opens multiple data streams for passing subdivided segments of a file simultaneously. Although the speed of each stream is same as in sequential streaming, the total time to move data in all streams can be significantly reduced compared to FTP transfer.
Striped data transfer a data objects is partitioned into a number of sections and each section is placed in an individual site in a data grid. When a user requests this piece of data, a data stream is created for each site in a data gird. When user requests this piece of data, data stream is created for each site, and all the sections of data objects ate transected simultaneously.

58. Grid Services and OGSA
Facilitate use and management of resources across distributed, heterogeneous environments
Deliver seamless QoS
Define open, published interfaces in order to provide interoperability of diverse resources
Exploit industry-standard integration technologies
Develop standards that achieve interoperability
Integrate, virtualize, and manage services and resources in a distributed, heterogeneous environment
Deliver functionality as loosely coupled, interacting services aligned with industry- accepted web service standards

59. OGSA services fall into seven broad areas, defined in terms of capabilities frequently required in a grid scenario. Figure shows the OGSA architecture. These services are summarized as follows:

60. OGSA services - seven broad areas
Infrastructure Services Refer to a set of common functionalities, such as naming, typically required by higher level services.
Execution Management Services Concerned with issues such as starting and managing tasks, including placement, provisioning, and life-cycle management. Tasks may range from simple j obs to complex workflows or composite services.
Data Management Services Provide functionality to move data to where it is needed, maintain replicated copies, run queries and updates, and transform data into new formats. These services must handle issues such as data consistency, persistency, and integrity. An OGSA data service is a web service that implements one or more of the base data interfaces to enable access to, and management of, data resources in a distributed environment. The three base interfaces, Da ta Access, Da ta Fa ctory, and Da ta Ma na gement, define basic operations for representing, accessing, creating, and managing data.

61. OGSA services - seven broad areas
Resource Management Services Provide management capabilities for grid resources: management of the resources themselves, management of the resources as grid components, and management of the OGSA infrastructure. For example, resources can be monitored, reserved, deployed, and configured as needed to meet application QoS requirements. I t also requires an information model (semantics) and data model (representation) of the grid resources and services.
Security Services Facilitate the enforcement of security-related policies within a (virtual) organization, and supports safe resource sharing. Authentication, authorization, and integrity assurance are essential functionalities provided by these services.

62. OGSA services - seven broad areas
Information Services Provide efficient production of, and access to, information about the grid and its constituent resources. The term “information” refers to dynamic data or events used for status monitoring; relatively static data used for discovery; and any data that is logged. Troubleshooting is j ust one of the possible uses for information provided by these services.
Self-Management Services Support service-level attainment for a set of services (or resources), with as much automation as possible, to reduce the costs and complexity of managing the system. These services are essential in addressing the increasing complexity of owning and operating an I T infrastructure.

63. References
Kai Hwang, Geoffery C. Fox and Jack J. Dongarra, “Distributed and Cloud Computing: Clusters, Grids, Clouds and the Future of Internet”, First Edition, Morgan Kaufman Publisher, an Imprint of Elsevier, 2012.