1. Evaluating Dynamic Services in Bioinformatics Maíra R. Rodrigues Michael Luck University of Southampton, UK Tenth International Workshop CIA 2006, Edinburgh

2. CIA 2006ECS - University of Southampton 2 Outline Bioinformatics Agents and Bioinformatics Model for Cooperative Interactions: Overview Requirements for Service Evaluation Evaluation Method Example Scenario Conclusion Future Work

3. CIA 2006ECS - University of Southampton 3 Bioinformatics Application of computer technology to manage and analyse biological data Bioinformatics Services Heterogeneous Locally and remotely used Continuous update Management and analysis of biological data and tools Suitability of an agent-based approach

4. CIA 2006ECS - University of Southampton 4 Bioinformatics Interrelated data Cooperative applications Participants request and provide services to each other Services free of charge Non-economic exchange of different types of tools and data Interactions are based on reciprocal relations

5. CIA 2006ECS - University of Southampton 5 Agents and Bioinformatics The agent-based approach: Agents provide and request bioinformatics services Existence of alternative providers Services are provided with different levels of quality (heterogeneity) Therefore.. Agents need to select service providers

6. CIA 2006ECS - University of Southampton 6 Agents for Interaction Agent-based applications in bioinformatics: Concerned with high-level management tasks Our concern: Model non-economic cooperative interactions Evaluation method for bioinformatics services to determine an agent’s satisfaction Guide agent’s decisions over service providers

7. CIA 2006ECS - University of Southampton 7 Model for Cooperative Interactions A1A1 A2A2 service Model non-economic cooperative interactions based on exchange values (Piaget 1973) satisfaction debt effort credit

8. CIA 2006ECS - University of Southampton 8 Model for Cooperative Interactions A1A1 A2A2 satisfaction debt effort credit Model non-economic cooperative interactions based on exchange values (Piaget 1973) A1A1 A2A2 credit satisfaction debt effort

9. CIA 2006ECS - University of Southampton 9 Model for Cooperative Interactions Exchange values result from the agent’s evaluation of the service Partner Selection (future interactions) Exchange Values Service Evaluation

10. CIA 2006ECS - University of Southampton 10 Model for Cooperative Interactions Exchange values result from the agent’s evaluation of the service Exchange values (Rodrigues, Luck 2005, 2006) Current work focus on service evaluation Partner Selection (future interactions) Exchange Values Service Evaluation

11. CIA 2006ECS - University of Southampton 11 Bio-Services are dynamic: Constant updates Regular behaviour, but Sensitive to different parameter configuration Evaluation requires Repeated evaluation Attach context information Evaluation of different aspects of the service Service Evaluation

12. CIA 2006ECS - University of Southampton 12 Evaluation method should address: Generality: apply to different types of bio- services and aspects of these services Continuity: repeat evaluation every time a service is received Consistency: compare evaluations made at different points in time Discriminated information: allow flexible decision-making by using evaluation of individual aspects or a global evaluation Service Evaluation

13. CIA 2006ECS - University of Southampton 13 Alternative Approaches Quantitative approaches Scoring or utility functions Objective values Precision, consistency, combination is straightforward Qualitative approaches: Classification rules (e.g., poor, good, excellent) Subjective values

14. CIA 2006ECS - University of Southampton 14 Evaluation Method Choose evaluation attributes for service examples: performance, quality, reliability, etc. For each attribute, associate result measures Pieces of information derived from service result that can determine the service utility in relation to an attribute (observed value). Static or dynamic measures (e.g., quality of interface and response time)

15. CIA 2006ECS - University of Southampton 15 Evaluation Method General evaluation function for evaluation attributes (utility): For a set of attributes A = {a 1,..,a i } U i = b c result measure for a i evaluation strictness 0 1 c UiUi

16. CIA 2006ECS - University of Southampton 16 Evaluation Process Before evaluation: Identify evaluation attributes for services and result measures for each attribute Repeat evaluation process every time a service is received Input is the service result and configuration used For each evaluation attribute a i Compute result measures Calculate evaluation U i Store evaluation Output is a set of evaluations (evaluation tuple)

17. CIA 2006ECS - University of Southampton 17 Evaluating Bio-Services Proteomics research Protein identification services Input: file (list of unknown peptides) Process: database + matching algorithm Output: list of proteins, peptides per protein Services: OMSSA, MASCOT, Tandem Local and Remote Heterogeneous results for same input data Sensitive to different input configurations Evaluation can be used as criterion for future selection

18. CIA 2006ECS - University of Southampton 18 Evaluation attributes: Sensitivity Capacity of matching related proteins Accuracy Capacity of identifying true matches Performance Time taken from input submission until result is received Evaluating Bio-Services

19. CIA 2006ECS - University of Southampton 19 Result measures (rm): Sensitivity Number of proteins Peptide ratio - peptides per protein Influence of input size Increasing utility input_size peptide_ratio x protein_number Evaluating Bio-Services rm =

20. CIA 2006ECS - University of Southampton 20 Accuracy Number of false positives Decreasing utility rm = false_positives Performance: Response time Influence of input size Decreasing utility response_time input_size Evaluating Bio-Services rm =

21. CIA 2006ECS - University of Southampton 21 Evaluation functions: U i = 0.5 rm Sensitivity (U 1 ): U 1 increases with peptide_ratio and protein_number Accuracy (U 2 ): U 2 decreases with false_positives Performance (U 3 ): U 3 decreases with response_time Evaluating Bio-Services

22. CIA 2006ECS - University of Southampton 22 Evaluating Bio-Services Practical evaluation: Same input spectra Two different configurations (C 1 and C 2 ) Evaluation of sensitivity Evaluation reflects different results for C 1 and C 2

23. CIA 2006ECS - University of Southampton 23 Evaluation of performance Again, evaluation reflects different results for C 1 and C 2 Evaluating Bio-Services

24. CIA 2006ECS - University of Southampton 24 Conclusions Present an evaluation method to be used by agents requesting dynamic services in bioinformatics Discussion of issues for efficient evaluation of these services, including Adoption of a repeated evaluation process Absolute evaluations Generation of individual and compatible evaluations Single evaluation must be calculated during selection

25. CIA 2006ECS - University of Southampton 25 Conclusions Show the application of the evaluation method for protein identification services Importance of dynamic (repeated) evaluation is shown through empirical results Provide more accurate information for agents that need to select services with dynamic characteristics

26. CIA 2006ECS - University of Southampton 26 Future Work Develop selection strategies that use and combine service evaluations Combination through objective and subjective values Probabilistic analysis of past evaluations Consider similarity between different service configurations Validate evaluation results with those of bioinformaticians

27. CIA 2006ECS - University of Southampton 27 Thank you

28. CIA 2006ECS - University of Southampton 28 References J. Piaget. Sociological Studies. Routlege, London, 1973. M. R. Rodrigues and M. Luck. Analysing partner selection through exchange values. In Jaime Sichman and Luis Antunes, editors, Multi-Agent-Based Simulation VI, volume 3891 of Lecture Notes in Artificial Intelligence, pages 24-40, Berlin Heidelberg, 2006a. Springer-Verlag. M. R. Rodrigues and M. Luck. Cooperative interactions: An exchange values model. In Coordination, Organization, Institutions and Norms in Agent Systems (COIN), ECAI Conference, Riva del Garda, Italy, August 2006b.