PrIMe PrIMe : Provenance Incorporating Methodology Steve Munroe The EU Grid Provenance Project University of Southampton UK

PrIMe 2 EU Grid Provenance Consortium University of Southampton –Luc Moreau, Steve Munroe, Sheng Jiang, Paul Groth, Simon Miles IBM UK (Project Coordinator) –John Ibbotson, Neil Hardman, Alexis Biller University of Wales, Cardiff –Omer Rana, Arnaud Contes, Vikas Deora Universitad Politecnica de Catalunya (UPC) –Steven Willmott, Javier Vazquez SZTAKI –Laszlo Varga, Arpad Andics German Aerospace –Andreas Schreiber, Guy Kloss, Frank Danneman

PrIMe 3 Overview of Talk Introducing Provenance Introducing PrIMe Stepping through PrIMe –Step 1. Provenance use cases –Step 2. Information items –Step 3. Identifying actors –Step 4. Actor interactions –Step 5. Knowledgeable actors –Step 6. Adaptations Evaluation Summary Conclusions

PrIMe 4 Provenance: dictionary definition Oxford English Dictionary: –the fact of coming from some particular source or quarter; origin, derivation –the history or pedigree of a work of art, manuscript, rare book, etc.; concretely, a record of the ultimate derivation and passage of an item through its various owners.

PrIMe 5 Provenance Definition Our definition of provenance in the context of applications for which process matters to end users:  The provenance of a piece of data is the process that led to that piece of data Our aim is to conceive a computer-based representation of provenance that allows us to perform useful analysis and reasoning to support our use cases We use the notion of Process Documentation, which is composed of p-assertions

PrIMe 6 Provenance Applications Aerospace engineering: maintain a historical record of design processes, up to 99 years. Organ transplant management: tracking of previous decisions, crucial to maximise the efficiency in matching and recovery rate of patients

PrIMe 7 Provenance (2) High Energy Physics: tracking, analysing, verifying data sets in the ATLAS Experiment of the Large Hadron Collider (CERN) Bioinformatics: verification and auditing of “experiments” (e.g. for drug approval)

PrIMe 8 Types of p-assertions (1) –Interaction p-assertion: is an assertion of the contents of a message by an actor that has sent or received that message I received M1, M4 I sent M2, M3

PrIMe 9 Types of p-assertions (2) –Relationship p-assertion: is an assertion, made by an actor, that describes how the actor obtained output data or the whole message sent in an interaction by applying some function to input data or messages from other interactions. M2 is in reply to M1 M3 is caused by M1 M2 is caused by M4 M3 = f1(M1) M2 = f2(M1,M4)

PrIMe 10 Types of p-assertions (3) –Actor state p-assertion: assertion made by an actor about its internal state in the context of a specific interaction I used sparc processor I used algorithm x version x.y.z

PrIMe 11 The Provenance Middleware Architecture

PrIMe 12 Introducing PrIMe A software engineering methodology for making applications provenance-aware

PrIMe 13 Introducing PrIMe: Key aims Provide software engineering guidelines for: identifying and expressing provenance use cases identifying the kinds of information items that are required to satisfy use cases identifying actors and the interactions between them in order to effect the recording of process documentation identifying the set of adaptations that integrate the provenance architecture with the application to expose the right kinds of information.

PrIMe 14 Introducing PrIMe: Overview Step 6. Adaptations Step 5. Knowledgeable Actors Application Structure Step 3. ActorsStep 4. Interactions Step 1. Use Cases Step 2. Information Items

PrIMe 15 Step 1: Provenance Use Cases Application Structure Step 1. Use Cases

PrIMe 16 Step 1: Provenance Use Cases We distinguish two types of provenance use case –A core provenance use case is a use case known when PrIMe is applied. –A future provenance use case is a use case that is not considered until after a process in the application is enacted but uses documentation of that process

PrIMe 17 Step 1: Provenance Use Cases Gathering Use Cases It is not always obvious to users what use cases they could expect the provenance middleware architecture to support. We provide a simple requirements elicitation process to help designers collect the core provenance use cases –Give definitions of provenance –Give examples of the general questions that can be answered using the architecture

PrIMe 18 Step 1: Provenance Use Cases Definition of Provenance The provenance of a result is the process that produced that result.

PrIMe 19 Step 1: Provenance Use Cases The OTM Application In the OTM application, use case questions relate to specific objects within the application, e.g.: –Recipients of organs –Organs –Organ Donors –Decisions

PrIMe 20 Step 1: Provenance Use Cases OTM Use Case Questions Below are questions that have been taken from the OTM application. –Retrieve data linked to all actions / events associated with a patient (recipient or donor) –What decisions were made for a particular case? –What is the medical analysis tree for a given organ? –Determine if any deviation took place from the standard workflow for a given organ

PrIMe 21 Step 1: Provenance Use Cases Eliciting Use Cases: We are looking to elicit use cases of the form: –(1) Actor A does something. –(2) Actor B does something else etc. –(3) Actor C determines the answer to a question about the provenance of data (such as a specific example of one of those above).

PrIMe 22 Step 1: Provenance Use Cases Elicitation Steps 3 Important steps: –Step (1) Describe something that already happens in the application. –Step (2) Describe a specific provenance- related use case question that cannot be answered (easily), but our functionality could help to achieve. –Step (3) Identify the relevant services required for answering the use case.

PrIMe 23 Step 1: Provenance Use Cases Example use case Donor A’s organs are screened for potential donation. What is the provenance of the donor organ’s diagnosis?

PrIMe 24 Step 1: Provenance Use Cases Form-Based Capture Donor A’s organs are screened for potential donation. What is the provenance of the donor’s organ diagnosis? Application DescriptionDonor A’s organs are screened for potential donation Use case questionWhat is the provenance of the donor’s organ diagnosis Relevant servicesUI, Donor data collector, EHCR, Testing Lab Relevant information itemsPID, Patient records, Blood analysis result, Analysis decision

PrIMe 25 Step 2: Information items Application Structure Step 1. Use Cases Step 2. Information Items

PrIMe 26 Step 2: Information Items Overview The kinds of information that will answer your use case May be one piece or many pieces of information –e.g. a given result, or a sequence of decisions For each use case, identify the information items required to satisfy the use case.

PrIMe 27 Step 2: Information Items Examples Information items may be : –Data items, i.e. the result of some calculation, decision (found in interactions or actor state). –Whole or part processes, e.g. the sequence of decisions that led to a donor’s organ being rejected for donation. –Relationships, e.g. what were the causal determinants of a given decision.

PrIMe 28 Step 2: Information Items Capture Information items are to be captured by process documentation, i.e. p-assertions –Data items: Interaction or actor state p- assertions –Processes: Interaction and relationship p- assertions

PrIMe 29 Step 3: Actors Step 3. Actors Step 1. Use Cases Step 2. Information Items Application Structure

PrIMe 30 Step 3: Actors Description An actor is an entity within the application that performs actions, e.g. Web Services, components, machines, people etc. and interacts with other actors. –One actor may be seen as being composed of other actors.

PrIMe 31 Step 3: Actors Roles in a provenance architecture Asserting Actors – assert p-assertions Recording Actors – record p-assertions Querying Actors – retrieve p-assertions Managing Actors – maintain provenance stores

PrIMe 32 Step 3: Actors Identification Heuristics Identify the components that receive information. E.g. a component/service in a workflow, a script command, the GUI/desktop application into which a user enters information. Identify the components that provide the information in each interaction. These could be, for example, a workflow engine, a script executor, a user.

PrIMe 33 Step 3: Actors OTM Example User interface Donor data collector Electronic health care records Testing laboratory Get Donor info (M1) Request p ID (M2) Return p ID (M3) Request blood test (M4) Return result (M5) Return result (M6)

PrIMe 34 Step 4: Interactions Application Structure Step 3. ActorsStep 4. Interactions Step 1. Use Cases Step 2. Information Items

PrIMe 35 Step 4: Interactions Information exchange

PrIMe 36 Step 3: Actors Information Message IDData itemReceiver ID M2 M4 M6 Q1 Pid r1 EHCR Testing Lab UI Donor data collector Sending Receiving Message IDData itemSender ID M1 M3 M5 q1 pid R1 UI EHCR Testing lab Actor : Donor data collector

PrIMe 37 Step 4: Interactions Tracking processes A common information item required for provenance use cases is the process to which documentation refers Interaction p-assertions Relationship p-assertions Tracers

PrIMe 38 Step 4: Interactions Session Tracer

PrIMe 39 Step 4: Interactions Tracer terminology A computational activity –Actors cooperating on some work Superiors –Any actor sending requests to other actors Inferiors –Any actor receiving requests from other actors Tasks –An independent computation within an actor, delimited by a request to the actor and a subsequent response from the actor

PrIMe 40 Step 4: Interactions Session Tracer Semantics Generation rule –An actor must generate a new session tracer at the start of each task and add the tracer to all requests within that task Propagation rule (to inferior) –An actor must add any session tracers received from a superior to all requests it makes to inferiors within the task started by the superior’s request Propagation rule (to superior) –An inferior must add the session tracers supplied by its superior to its response to its superior

PrIMe 41 Step 4: Interactions Other Tracers Other application specific tracers possible –e.g. In the medical domain, a tracer could be used to identify all interactions belonging to a particular case.

PrIMe 42 Step 5: Knowledgeable Actors Step 5. Knowledgeable Actors Application Structure Step 3. ActorsStep 4. Interactions Step 1. Use Cases Step 2. Information Items

PrIMe 43 Step 5: Knowledgeable Actors Knowledgeable actors have access to Information items Sometimes, for a given information item, a knowledgeable actor cannot be found Further decomposition might be necessary or, New actors may need to be introduced (Step 6)

PrIMe 44 Step 5: Knowledgeable Actors Who knows what? Hospital EHCRS Testing lab

PrIMe 45 Step 5: Knowledgeable Actors OTM Example User interface Donor data collector Electronic health care records Testing laboratory Hospital

PrIMe 46 Decomposing the Hospital

PrIMe 47 Step 3,4,5: Knowledgeable Actors Repeat as necessary –Step 3: Identify actors –Step 4: Identify interactions –Step 5: Identify knowledgeable actors

PrIMe 48 Step 6: Adaptations Step 6. Adaptations Step 5. Knowledgeable Actors Application Structure Step 3. ActorsStep 4. Interactions Step 1. Use Cases Step 2. Information Items

PrIMe 49 Step 6: Adaptations Modifying actors A non-knowledgeable actor may be modified so that it gains access to information items not currently available to itself or other actors in the system.

PrIMe 50 Step 6: Adaptations Actor Introduction A new actor can be introduced to the application to help in the answering of use cases

PrIMe 51 Step 6: Adaptations Interaction Extension An interaction in the application can be extended to exchange more information between a knowledgeable actor and a non- knowledgeable actor, making the latter knowledgeable. Actor Before Actor After a a,b

PrIMe 52 Step 6: Adaptations Interaction Introduction A new interaction between actors can be introduced into the application in which a knowledgeable actor sends the information item to another actor, which then becomes knowledgeable. Actor Before Actor After b

PrIMe 53 Step 6: Adaptations Provenance Functionality The provenance wrapper exposes an actor’s input and output data, relationships and aspects of the actor’s state.

PrIMe 54 Step 6: Adaptations The Client Side Library A collection of functions –To allow provenance-aware applications to communicate with provenance store services –An implementation of the actor side library should contain at least one of the query library, the record library and the management library –Helps application developers follow architecture rules

PrIMe 55 Step 6: Adaptations CSL Layered Approach Client Side Library Applications Provenance Store Server Application API Utilities Server API

PrIMe 56 Recording Provenance

PrIMe 57 Evaluation Protein compressibility experiment 10% overhead for asynchronous recording

PrIMe 58 Summarising PrIMe Step 1: Identify the provenance use cases Step 2: Identify relevant information items –Step 3: Identify actors –Step 4: Identify interactions –Step 5: Identify knowledgeable actors Step 6: Make necessary adaptations Granularity

PrIMe 59 Conclusions PrIMe provides a clear and easy guide to make applications provenance-aware Crucial in the adoption of the Provenance Middleware Architecture

PrIMe 60 Questions? Steve Munroe