EThOSnet Repositories and Web Services Workshop 2 nd June2009 Richard Green

EThOSnet BL Workshop 2 nd June 2009 … an executable business process that may interact with both internal and external Web services. Orchestration describes how Web services can interact at the message level, including the business logic and execution order of the interactions. These interactions may span applications and/or organizations, and result in a long-lived, transactional process. With orchestration, the process is always controlled from the perspective of one of the business parties. Chris Peltz, Hewlett-Packard Co,

EThOSnet BL Workshop 2 nd June 2009 Need some way of structuring the flow from one Web Service to the next Needs to support things like loops, conditional branching, … Need some form of language to describe the process A formal language such as BPEL? Something lighter-weight akin to a scripting language? …. ? 3

EThOSnet BL Workshop 2 nd June 2009 Hull has been developing workflows using BPEL for the last four years (Business Process Execution Language – an open standard) Used in conjunction with SOAP Web Services during the RepoMMan and REMAP projects JISC-funded projects &

EThOSnet BL Workshop 2 nd June 2009 In 2005 Hull (and JISC) had an interest in using BPEL within a Service Oriented Architecture BPEL (then) available in an Open Source engine from Active Endpoints Note: the engine is now called ActiveVOS Good fit with Fedoras (then) SOAP Web Services interface (REST now available too) 5

EThOSnet BL Workshop 2 nd June count($getCollectionItemsResponse/itemList/itemPID) itemList[ $counter ]/itemPID pid '' asOfDateTime itemList[$counter]/itemPID types:itemsRef[$counter]/types:objectPID itemList[$counter]/isCollection types:itemsRef[$counter]/types:isCollection objectProfile/objLabel types:itemsRef[$counter]/types:label objectProfile/objLastModDate types:itemsRef[$counter]/types:lastModified contains(string($getCollectionItemsResponse/itemList[$counter]/isCollection), 'true') '' types:itemsRef[$counter]/types:mimeType itemList[$counter]/itemPID pid file dsID '' asOfDateTime datastream/MIMEType types:itemsRef[$counter]/types:mimeType 6

EThOSnet BL Workshop 2 nd June Cons: verbose, fiddly, syntactically demanding, soul destroying, …. Pros (given a good graphical design interface): powerful, flexible, relatively quick to write, test and edit… Each node in the tree is an activity (for each, assign, get, if, etc) for which you provide the parameters Note: the for each loop depicted here results in the code on the previous slide

EThOSnet BL Workshop 2 nd June 2009 The REMAP tool (son of RepoMMan) uses BPEL-orchestrated Web Services to allow a user to interact with the institutional repository Each component Web Service can be used and re-used in multiple contexts given appropriate granularity 8

EThOSnet BL Workshop 2 nd June 2009 Consider a user copying a file from their computer to their private repository space 9 They browse their computer at the left and upload the file to their repository space, represented at the right. Lots of stages (Web Services) involved under the lid

EThOSnet BL Workshop 2 nd June

EThOSnet BL Workshop 2 nd June 2009 The user can (optionally) publish a file to the institutional repository. The tool provides a context sensitive wizard. The process is moderated through an accession queue. Take the example of a thesis (ETD) 11

EThOSnet BL Workshop 2 nd June 2009 Lots of Web Services involved, but the two outlined in red could easily be remote services 12

EThOSnet BL Workshop 2 nd June 2009 Uses a tool developed in the US (University of California) to extract descriptive metadata and exposed as a Web Service Implemented locally in Hull, but in terms of functionality could just as well have been called from the US. Local implementation cuts down data transfer requirements Uses another US tool (Harvard) to extract technical metadata, again we have exposed it as a Web Service ditto The final publish stage of the process invokes DROID developed by The National Archives and exposed in Hull as a Web Service… Ditto, except that TNA is closer to home … which in turn invokes PRONOM hosted at TNA Not much data to transfer, takes advantage of frequent updates to the service by TNA 13

EThOSnet BL Workshop 2 nd June 2009 Orchestrating a large collection of Web Services Fedora Home-grown Services wrapped around other peoples tools Remote services …builds us an object in the repository 14

EThOSnet BL Workshop 2 nd June 2009 This has described one orchestrated workflow. Hull pursuing BPEL for now although the Active Endpoints Open Source BPEL engine is no longer being developed by Active VOS 15

EThOSnet BL Workshop 2 nd June 2009 BPEL (and similar approaches such as jBPM) are formal ways of orchestrating a workflow There are those who consider them unnecessarily heavyweight Other approaches are possible. Hulls partners in the Hydra Project are looking at two… 16

EThOSnet BL Workshop 2 nd June 2009 The Hydra Project is building a flexible, configurable, end-to-end workflow solution (add, edit, delete, search, discover, orchestrate workflow components) based on Fedoras REST services and Ruby for rapid agile development Part of Hydra will be a Lego set of Web Services for the Fedora community that go beyond the basic set provided by Fedora The three Hydra university partners are each developing a different approach to workflow orchestration 17

EThOSnet BL Workshop 2 nd June 2009 Hull: BPEL (at least for now) The withdrawn support for the Open Source BPEL engine is potentially a problem University of Virginia: Lightweight scripting language under consideration Stanford: Robots based on Ruby scripts process a workflow set out in the Fedora object itself. One autonomous robot per process 18 A workflow datastream in each object describes processing requirements and status <process name="google-download" status="exception message="Item for barcode not found" attempts="3" /> After Lynn McRae, SULAIR, Stanford

EThOSnet BL Workshop 2 nd June 2009 Orchestrated Web Services can be a very flexible development technique Web Services may be local or remote; yours or someone elses Web Services are modular and may be (re)used in multiple contexts Orchestration is potentially achieved by a variety of means ranging from formal languages to simpler scripts 19

EThOSnet BL Workshop 2 nd June 2009 Fedora: fedora-commons.org Hull repository: edocs.hull.ac.uk REMAP: Hydra: fedora-commons.org/confluence/display/hydra Active Endpoints: 20