1. DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality Interoperability Approaches, case studies and open issues DL.org Quality Working Group Rome, 28 th May 2010

2. DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality comprehensive models Fuhr et al., 2001 Digital Libraries: A Generic Classification and Evaluation Scheme

3. DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality comprehensive models Goncalves et al., 2006 What is a good digital library? A quality model for digital libraries

4. DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality comprehensive models Zhang, 2010 Holistic DL evaluation model

5. DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality comprehensive models Candela et al., The DELOS RM Quality concept map, 2008 Annotations by the DL.org Quality WG

6. DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality Quality is subjective Quality is dynamic Quality is vague Quality needs policies …

7. DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality WG meeting results Our motivating scenario: consider that representatives of two (or more) DLs have a round table to negotiate a service level agreement (SLA) defining their interoperability requirements and for this establish a quality threshold that each individual DL has to meet or exceed; “Quality” would provide transparent qualitative or quantitative parameters for defining the threshold Our approach is practical: Quality Interoperability Survey, Quality scenarios The Cookbook TOC The interoperability scenarios 1&2

8. DL.org All WGs Meetings, Rome, 26-28 May 2010 Quality Interoperability Survey Survey Pilot questionnaire + results Simplification and improvement Disambiguation (Glossary) Collection strategy Data analysis and interpretation Results expected by June 2010 Best practices

9. DL.org All WGs Meetings, Rome, 26-28 May 2010 The Cookbook Definitions of quality Context: how the Quality WG faced the investigation challenges Consistent terminology Introduction

10. DL.org All WGs Meetings, Rome, 26-28 May 2010 The Cookbook Looking at Web solutions at a technical level FRONT END: Web Interfaces (usability, accessibility), eg. W3C BACK END: Web services ontologies (QoS), eg. FIPA, WS-QoS, MOQ, DAML-QoS

11. DL.org All WGs Meetings, Rome, 26-28 May 2010 The Cookbook Quality insurance procedures for specific DL systems Semantic and organisational levels Document repositories (eg. DINI, DRIVER) Research data archives (eg. Data Seal of Approval) Preservation systems (eg. TRAC, DRAMBORA) ↓ Template for comparison within each class

12. DL.org All WGs Meetings, Rome, 26-28 May 2010 The Cookbook – Case studies Template AspectDINI Certificate DRIVER Guidelines Explicit quality policy for protocol and metadata implementationYes Explicit policy for operations (personell, support etc.)YesNo Personal quality check (questionaire, on-site review)YesNo Intellectual quality check (remote)Yes Automatic self validationNoYes Organized through sustainable OrganisationDINICOAR Explicit branding when checkedYesNo Translation in English, Spanish, Portuguese, JapaneseNoYes Green and GoldYesNo Strictly full-text orientedYes

13. DL.org All WGs Meetings, Rome, 26-28 May 2010 The Cookbook Best practices from the professional community The results of the Survey will be included as best practices from the professional community. We are aware that quality is subjective, that we are dealing also with two “primitive” interoperability challenges 1. researchers vs professionals 2. different disciplines involved But we want to know from DLs people!

14. DL.org All WGs Meetings, Rome, 26-28 May 2010 The Cookbook The Quality WG checklist Light-weight document – based on the Quality Core Model parameters - with practical recommendations from the DL.org Quality WG based on the QCM parameters and the Quality Interoperability Survey results

15. DL.org All WGs Meetings, Rome, 26-28 May 2010 Interoperability Scenarios Scenario no. 1 – Quality and Functionality issues  Metadata Evaluation  Formats quality  Geographic referencing (Content) and related pinpointing issues (Functionality)  Controlled vocabulary + related Functionalities issues (search, browsing, etc)  Multilingualism  Quality of the annotations (User profiling, User authentication)  Sources evaluation, i.e. Provenance

16. DL.org All WGs Meetings, Rome, 26-28 May 2010 Interoperability Scenarios Scenario no. 1 – Policy issues  Copyright issues  Reuse issues  Content policy issues Possible solutions: change the copyright policy, use creative commons license Suggestions: review the scenario and make it consistent (logical incoherencies on map reproduction, restored video, 3D Model regeneration), less chaotic and more realistic

17. DL.org All WGs Meetings, Rome, 26-28 May 2010 Interoperability Scenarios Scenario no. 2 – Quality issues Merged query results Machine translations for supporting two languages Articles quality (one has editorial reviews the other has not) Mark-up of content Metadata evaluation: Accuracy and Completeness System performance Technical correctness

18. DL.org All WGs Meetings, Rome, 26-28 May 2010 Interoperability Scenarios Scenario no. 2 - Approaches Two localizations, one for each country, each supporting one language for content from both systems (very difficult!) Functionalities of both source DLs (i.e. union of), including: access control, search, results views Metadata quality: solved selecting targets, customizing search screen based on targets e.g. which common metadata fields they support Search QoS: solution in the user interface design to indicate how far along the search is and cluster the res ults Ownership of images: search results must contain author, photographer, agency and rights metadata so the user can see if they need to pay, who they pay, and what rights they get

19. DL.org All WGs Meetings, Rome, 26-28 May 2010 Some conclusive thoughts Quality: dynamic, subjective, not completely definable Provenance = the resource story = how to establish quality DL systems and their implementation teams will always struggle with quality, but users will have the last word

20. DL.org All WGs Meetings, Rome, 26-28 May 2010 THANK YOU