Flexible Service Composition Adam Barker School of Informatics University of Edinburgh Robert G. Mann Institute for Astronomy University of Edinburgh

Introduction Service Oriented Architectures –Distributed computing platform targeted at the web –Define a standard way to perform program to program interaction –Can tie together any OS, application, data store, programming language, device etc. –Defined using: XML, SOAP, WSDL etc. A Service Oriented World –Wide spread adoption and interest in simple, vanilla web service standards –This standardised architecture is unlikely to change in the forcible future Technologies need to fit around and consume these existing standards

Scientific Workflow Most workflow engines focus on Business Process Modelling A Scientific Workflow captures a series of analytical steps which describe a computational experiment –In order to verify a hypothesis etc. Scientific Workflow has an extra set of requirements: –Rapid prototyping of experiments –User Interaction with the scientist –Reliability and Fault Tolerant execution –Transparent access to resources –Repeatability, Smart re-runs and parameterisation –Provenance information, presentation of the results –Control Flow vs. Data Flow

Motivation Few languages which deal with the flexible, knowledge acquisition and discovery processes found in the sciences – myGrid, Kepler, ICENI frameworks allow statically defined, pre-designed/pre-planned workflows to be executed by a centralised workflow engine The following science use-case serves as a counter example of coordination which is difficult to achieve by existing systems –Large Synoptic Survey Telescope (LSST) –Time-Domain astronomy

Science Use Case Background Current observatories are able to look very deeply at very small sections of sky, not likely to catch transient events: –Observatories always looking somewhere else –Small field of vision means that an impractically large number of separate observations are required to map the entire night sky –Observing facilities are scarce Observations of change in the universe are difficult to obtain

Science Use Case Background Automated Classification –For known classes of object –LSST is a first attempt at time-domain astronomy Likely to discover new (previously undetected) classes of object –Always data which the automated stage cannot classify Typically most of these will be junk, however this may only be revealed by comparisons with other detections made on the same night

Science Use Case

WorkFlow Requirements Complex coordination behaviour –Data is inherently distributed –Resources are scarce Requires selection and negotiation –Large quantities of data Requires Autonomous, Intelligent software Human in the loop only when needed, e.g. specialist –Workflow specification cannot be fully predicted at design-time Difficult to classify data type or quantity in advance Component choice, e.g. web service, database etc. –(semi) Flexible, Dynamic, runtime composition In a decentralised, peer-to-peer environment

A convergence of Interests

Workflow based on Interaction Protocols Interaction Protocols –Collection of conventions that allow cooperation between agents in an open MultiAgent System –A recipe for communication –Defines if and when agents communicate –Order and kind of messages relating to a certain domain Multi Agent Service Composition (MASC) –Agent-Based Workflow language and framework based on Interaction Protocols –Decentralised, peer-to-peer architecture –Aimed at Scientific Workflow composition In order to solve the motivating science use-case –Discuss the language and framework in the following slides

MASC: Scenes Scene –Bounded space in which a group of agents interact on a single shared task –Divide a large, complex protocol into manageable chunks –Scenes cannot begin execution until all agents have reference to the protocol S := scene (id s, {ip}, {op}, {R}, {A}) ip := inport (id s :id pin, T) op := outport (id s :id pout,T )

MASC: Roles Role –A role type allows an engineer to specify a pattern of behaviour which an agent can adopt Many agents can adopt the same role type Roles are defined as a set of methods {M} –Methods are constructed from operations and actions A := agent (id a, id r, Φ (k) ) R := {id r, config (k), {M}} M := method id m (Φ (k) ) = op Φ:= v:T, _, c:T T:= XML Data Types

MASC: Operation Set op:= action | op 1 then op 2 | op 1 or op 2 | op 1 par op 2 | waitfor op 1 timeout op 2 | invoke id m (Φ (k) )

MASC: Action Set action := empty (empty) | p(Φ (k) ) agent ( id a, id r ) (send) | p(Φ (k) ) multicast (id r ) (multicast) | p(Φ (k) ) user () (user send) | p(Φ (k) ) agent (id a, id r ) (receive) | p(Φ (k) ) user () (user receive) | Φ (k) = portread (id pin ) (port read) | portwrite (id pout, Φ (k) ) (port write) | Φ (k) = p(Φ (l) ) fault Φ (m) (decision) | Φ (k) = service (ws, Φ (l) ) fault Φ (m) (service)

MASC: Decision Procedures Decision Procedures –Connect the protocol code (describing the interaction model) to an agents internal reasoning model –Each agent interacting within a scene references a set of decision procedures Implemented as a set of methods inside a reasoning web service Throughout protocol execution agents can invoke methods on its reasoning web service Does not sacrifice the autonomy of the agent Each agent can subscribe to their own reasoning model (BDI etc.) Up to the agent engineer to specify these methods Allowing a personalised strategy within the interaction protocol

MASC: Service Enactment External Web Services –Agents consume external web services, acting as a proxy to their execution Extra level of abstraction –Allows agents to consume the passive Service Oriented Architecture found in Internet and Grid Systems –Can be hard coded in the interaction protocol or determined at runtime e.g. from a broker or message exchange

MASC: Execution

MASC: Dataflow P := protocol (id p,{S}, link (L) * ) L := source sink + source/sink: –Web Service –Application –User interaction –File reading/writing High level experiment composition

MASC: Layers of Abstraction

The MASC Framework Implementation: –Full Java implementation Built using the Sun Java Web Services Development Pack (JWSDP) 2.0 –Agents can be executed locally: as a closely coupled system or as distributed processes –SOAP messaging –XML Language specification – Application –Requirements based on Real Science use cases –Applied to the UK e-Science project, AstroGrid

Advantages Standard MAS Arguments –Inherently a Distributed Peer-to-Peer System No centralised server No central point of failure Allows engineering in an open environment Interactions that are too complex, or simply cannot be specified at design-time –Agents act as a proxy to the services which are being coordinated, extra level of abstraction –Reactive agents, runtime decision and coordination

Advantages Inter-operability –Infrastructure independent: Interaction model always remains a layer above any specific middleware or OS –The web services being invoked require no modification before taking part in the interaction –Low engineering requirement Layer of s/w to translate protocol steps Reasoning Models –Protocols do not sacrifice the autonomy of the agents –Each agent can subscribe to their own reasoning model

Conclusions Internet and Grid Systems are filled with passive objects (services) Agency paradigm offers a way of programming autonomous, social and active components which consume this SOA Language provides a way of applying the principles and well understood concepts of agency to the web service composition problem Framework fits in with existing standards AstroGrid provides a live science test bed

Questions Thanks for listening