USC Viterbi School of Engineering Web and Grid Services Slides taken from a variety of sources: GT4 tutorial, by Borja Sotomayor International Summer school on Grid Computing 2007m by Malcolm Atkinson eu.org/issgc07/index.cfmhttp:// eu.org/issgc07/index.cfm HP introduction to WSRF by Sanjay Dahiya

USC Viterbi School of Engineering Service Oriented Architecture Web Services WS-RF Globus implementation of WS-RF OGSA-DAI

USC Viterbi School of Engineering Service Oriented Architectures: Three Components Registries Service Consumers Services Register an available service Send name & description

USC Viterbi School of Engineering Three Components Registries Service Consumers Services Request a service Send a description

USC Viterbi School of Engineering Three Components Registries Service Consumers Services Set (possibly empty) of matching services

USC Viterbi School of Engineering Three Components Registries Service Consumers Services Request service operation

USC Viterbi School of Engineering Three Components Registries Service Consumers Services Return result or Error

USC Viterbi School of Engineering Composed behaviour Services are themselves consumers –They may compose and wrap other services The registry is itself a consumer A federation of registries may deal with registry services reliability & performance Observer services may report on quality of services and help with diagnostics Agreements between services may be set up –Service-Level Agreements –Permitting sustained interaction

USC Viterbi School of Engineering Web Services

USC Viterbi School of Engineering Web Services Enable communication between machines Provide any number of functionality Can be found and invoked Self-describing—tell you how they can be invoked

USC Viterbi School of Engineering Web Services are platform-independent and language- independent, since they use standard XML languages Most Web Services use HTTP for transmitting messages (such as the service request and response) –Good for going through firewalls Issues: –Overhead: XML is not as efficient for data transmission as using a proprietary binary code. What you win in portability, you lose in efficiency. –Limited in functionality, no explicit state management or lifecycle management Web Services

USC Viterbi School of Engineering Typical Service Invocation

USC Viterbi School of Engineering The Web Services architecture

USC Viterbi School of Engineering A typical web service invocation Whenever a client needs to communicate with a service, it calls a client stub The client stub will turn this 'local invocation' into a proper SOAP request The SOAP request is sent over a network using the HTTP protocol. The server receives the SOAP requests and hands it to the server stub which invokes the service implementation

USC Viterbi School of Engineering Web Server Web service: software that exposes a set of operations SOAP engine: knows how to handle SOAP requests and responses-- Apache Axis Application server: provides a space for applications that multiple clients can access—a container--Jakarta Tomcat server Http server: knows how to handle HTTP messages Sometimes a container is described as: SOAP engine + application server + HTTP server

USC Viterbi School of Engineering Limitations of Web Services No explicit state management or lifecycle management Web services are usually stateless Many services do not need state

USC Viterbi School of Engineering Adding state to services Need to allow clients to access appropriate state

USC Viterbi School of Engineering “Stateless” vs. “Stateful” Services Without state, how does client: –Determine what happened (success/failure)? –Find out how many files completed? –Receive updates when interesting events arise? –Terminate a request? Few useful services are truly “stateless”, but WS interfaces alone do not provide built-in support for state Client FileTransfer Service move (A to B) move

USC Viterbi School of Engineering Open Grid Services Architecture

USC Viterbi School of Engineering Before OGSA Grid services before OGSA –Resource management (Globus GRAM) –Resource discovery (Globus MDS) –Data Management (Globus GridFTP, RLS) –Security All had –Different mechanism for access –Different mechanism for discovery –Different mechanisms for management

USC Viterbi School of Engineering OGSA Brings “order” to distributed services Promotes “open” standards: defined in GGF (now OGF), OASIS Enables Virtualization –Encapsulation behind a common interface of diverse implementations Allows the composition of lower-level services to form more sophisticated services Defines common behaviors that all services must have: –Naming –Lifetime management –State management –Notification

USC Viterbi School of Engineering

WSRF

USC Viterbi School of Engineering FileTransferService (without WSRF) Developer reinvents wheel for each new service –Custom management and identification of state: transferID –Custom operations to inspect state synchronously (whatHappen) and asynchronously (tellMeWhen) –Custom lifetime operation (cancel) Client FileTransfer Service move (A to B) : transferID move state whatHappen tellMeWhen cancel

USC Viterbi School of Engineering WSRF in a Nutshell Service State representation –Resource –Resource Property State identification –Endpoint Reference State Interfaces –GetRP, QueryRPs, GetMultipleRPs, SetRP Lifetime Interfaces –SetTerminationTime –ImmediateDestruction Notification Interfaces –Subscribe –Notify ServiceGroups RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR

USC Viterbi School of Engineering FileTransferService (w/ WSRF) Developer specifies custom method to createResource and leaves the rest to WSRF standards: –State exposed as Resource + Resource Properties and identified by Endpoint Reference (EPR) –State inspected by standard interfaces (GetRP, QueryRPs) –Lifetime management by standard interfaces (Destroy) Client FileTransferService createResource (A to B) : EPR createResource RPs Transfer getRP queryRPs destroy

USC Viterbi School of Engineering Grid Infrastructure: Open Standards Web services (WSDL, SOAP, WS-Security, WS-ReliableMessaging, …) WS-Resource Framework & WS-Notification* (Resource identity, lifetime, inspection, subscription, …) WS-Agreement (Agreement negotiation) WS Distributed Management (Lifecycle, monitoring, …) Applications of the framework (Compute, network, storage provisioning, job reservation & submission, data management, application service QoS, …)

USC Viterbi School of Engineering Endpoint References WS-Addressing specification An Endpoint Reference (EPR) points to a web service by including a URL. For WSRF, can include ReferenceProperties that identify a resource. Can also contain other information –Security –Renewable Reference information C

USC Viterbi School of Engineering WS-Resource Properties Each resource has a Resource Properties document. Resource Properties document is referred in service portType in WSDL. Defined in XML schema. Each element in the Resource Properties document is a Resource Property (RP). Resource properties can be queried using multiple query dialects. Independent of back-end implementation.

USC Viterbi School of Engineering Accessing Resource Properties Pull –Client can query the RP document, using query engines. GetResourceProperty GetMultipleResourceProperties QueryResourceProperties Push –Allows services to send changes in their resources’ RPs to interested parties. WS-Notification

USC Viterbi School of Engineering WS-Notification Subscriber indicates interest in a particular “Topic” by issuing a “subscribe” request Broker (intermediary) permits decoupling Publisher and Subscriber “Subscriptions” are WS-Resources Publisher need NOT be a Web Service Notification may be “triggered” by: –WS Resource Property value changes –Other “situations” Broker examines current subscriptions Brokers may –“Transform” or “interpret” topics –Federate to provide scalability

USC Viterbi School of Engineering WS-Notification Characteristics of WS-Notification: –Web services integration of traditional enterprise publish/subscribe messaging patterns Composes with other Web services technologies Facilitates integration between different messaging middleware environments –Standardizes the role of Brokers, Publishers, Subscribers and Consumers –Provides two forms of publish/subscribe: direct publishing and brokered publishing –Standardizes Web service message exchanges for publishing, subscribing and notification delivery –Defines XML model of Topics and TopicSpaces to categorize and organize notification messsages

USC Viterbi School of Engineering WS-Resource Lifetime Creating new resources. Destroying old resources. –Immediate destruction. –Scheduled destruction using termination time. Soft-state lifetime management –Lifetime extension. Example : –jobs in a batch submission system could be represented as resources –submitting a new job causes a new resource to be created –when the job is completed, the resource is destroyed.

USC Viterbi School of Engineering WS-ServiceGroups Service groups maintain information about collection of services or resources. ServiceGroupRegistration –Add new members to group using WS invocation. Represent service groups as resources. MembershipContentRules –Imposes restrictions on services that can become part of service group like implementing an interface. WS-Notifications for service group changes. For example – Resource registries etc

USC Viterbi School of Engineering WS-BaseFaults XML based fault transmission. Associated with an operation in WSDL. Includes standard datatypes for transmitting webservice faults –Originator, Timestamp etc.. Example :

USC Viterbi School of Engineering WS-Renewable References End Point References can become stale –Service might move to a different host Provides a way to take a stale reference and try to get a fresh one. Implemented as a decoration of WS-Addressing, adding some policy information to it.

USC Viterbi School of Engineering Globus Toolkit implementation of WSRF

USC Viterbi School of Engineering Data Mgmt Security Common Runtime Execution Mgmt Info Services GridFTP Authentication Authorization Reliable File Transfer Data Access & Integration Grid Resource Allocation & Management Index Community Authorization Data Replication Community Scheduling Framework Delegation Replica Location Trigger Java Runtime C Runtime Python Runtime WebMDS Workspace Management Grid Telecontrol Protocol Globus Toolkit v4 Credential Mgmt Globus Toolkit: Open Source Grid Infrastructure Tools for building WSRF services

USC Viterbi School of Engineering GT4 WS Core in a Nutshell RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR Implementation of WSRF: Resources, EndpointReferences, ResourceProperties Operation Providers: pre-build implementations of WSRF operations Notification implementation: Topics, TopicSet, Embedded Notification Consumer service

USC Viterbi School of Engineering GT4 WS Core in a Nutshell RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome ResourceHome: The home “owns” the Resource instances in the service SingletonResourceHome: manages single instance of Resource ServiceResourceHome: for services that support a single Resource instance ResourceHomeImpl: manages multiple Resource instances. Supports resources with in- memory state and resources with persistent (on disk) state

USC Viterbi School of Engineering Service Container GT4 WS Core in a Nutshell RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome Service Container: host multiple services in container; one JVM process …more details: based on AXIS service container, processes SOAP messages

USC Viterbi School of Engineering Service Container GT4 WS Core in a Nutshell RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome Secure Communication: Transport, Message, Conversation (Transport demonstrates best performance) PIP PDP Configurable Security Policies: Policy Information Points (PIPs), Policy Decision Points (PDP) -- chained Example authorization PDPs: GridMap, SAML implementations

USC Viterbi School of Engineering Service Container GT4 WS Core in a Nutshell RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome PIP PDP WorkManagerDB Conn Pool JNDI Directory WorkManager: “thread pool”, site independent “work” manager Apache Database Connection Pool library (JDBC “DataSource” implementation) JNDI Directory: manages internal, shared objects (ResourceHomes, WorkManager, Configuration objects,…)

USC Viterbi School of Engineering Apache Tomcat Service Container GT4 WS Core in a Nutshell RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome RPs Resource Service GetRP GetMultRPs SetRP QueryRPs Subscribe SetTermTime Destroy EPR ResourceHome PIP PDP WorkManagerDB Conn Pool JNDI Directory Deploy Service Container “standalone” or within Apache Tomcat

USC Viterbi School of Engineering Relationship Between OGSA, GT4, WSRF & Web Services

USC Viterbi School of Engineering Dealing with Data OGSA-DAI

USC Viterbi School of Engineering OGSA-DAI An extensible framework accessed via web services that executes data-centric workflows involving heterogeneous data resources for the purposes of data access, integration, transformation and delivery within a Grid and is intended as a toolkit for building higher-level application-specific data services

USC Viterbi School of Engineering Motivation Grid is about sharing resources OGSA-DAI is about sharing structured data resources

USC Viterbi School of Engineering Sharing data via website download ZIP up data and put it on a website Pros –Easy distribution for providers –Easy access for consumers Cons –Consumers have to download all the data –Consumers have to load data into local databases to use it –Static snapshot –Security

USC Viterbi School of Engineering Sharing data via direct access Providers tell consumers –Database URL – mycomputer.epcc.ed.ac.uk:3306 –Username – userID –Password – password Pros –Consumers have direct access, so it should be faster Cons –Firewall issues –User and password management is hard –No consistent security model –Hard to use in grid/web service workflows –Continued on next slide…

USC Viterbi School of Engineering Sharing data via direct access Cons (continued) –No server-side layer in which to standardize database heterogeneities –Client needs to know, and have installed, correct driver for the database. –Different drivers for Java, C#, C++, Fortran etc. –Totally different API for different database types, e.g. JDBC for Relational, XMLDB for XML, Lucene for indexed files.

USC Viterbi School of Engineering Manipulate data using domain-specific operations, e.g. –Book findByISBN(ISBN) –List findByAuthor(Author) –List findByKeyword(Word) Pros –Fits with grid/web service approach –Abstraction hides back-end database details –Web services are programming language neutral –Operations likely to map well to authorization policies Domain-specific web services

USC Viterbi School of Engineering Cons –Slower than direct access Web service layer SOAP transport overhead – especially for large result sets –Domain-specific API prevents use of generic data exploration, mining and manipulation tools Domain-specific web services

USC Viterbi School of Engineering OGSA-DAI generic web services Manipulate data using OGSA-DAI’s generic web services Clients sees the data in its ‘raw’ format, e.g. –Tables, columns, rows for relational data –Collections, elements etc. for XML data Clients can obtain the schema of the data Clients send queries in appropriate query language, e.g. SQL, XPath

USC Viterbi School of Engineering OGSA-DAI Pros: –Fits with grid/web service approach. –Web services are programming language neutral. –Access to schema and raw data supports generic tools. Cons: –Slower than direct connection mainly due to SOAP overhead. –One more layer between client and data –Data not transferred in efficient binary format.

USC Viterbi School of Engineering Reducing the SOAP effect – workflows