1. Towards a Reference Quality Model for Digital Libraries Maristella Agosti Nicola Ferro Edward A. Fox Marcos André Gonçalves Bárbara Lagoeiro Moreira

2. Goals Discuss the importance of defining a Reference Quality Model for Digital Libraries. Present current approaches for DL quality evaluation Define and discuss the necessary steps to achieve the Reference Quality Model.

3. Why a Reference Quality Model for Digital Libraries? To capture the notion of quality in DL and define quality measures for digital libraries; To detect problems in the system and obtain information to fix them; To follow the evolution of systems and their several components (e.g., collections, catalogs, services);

4. Why a Reference Quality Model for Digital Libraries? To evaluate content to be inserted in the system and check if their quality is compatible with content already in the system; obeying certain constraints, to compare two or more systems, with regards to some of their components.

5. Approaches for DL Quality Evaluation > Broad Studies and Conceptual Frameworks [Saracevic and Covi 2000] and [Saracevic 2000]: evaluation challenges and requirements are enumerated and an evaluation conceptual framework is suggested. [Saracevic 2004] provides an overview of the work on DL evaluation, analyzing the constructs that were evaluated, the context in which the evaluations were conducted, and the criteria that were chosen as a basis for evaluation.

6. [Fuhr et al. 2001] defines a DL quality conceptual model based on four dimensions: data & collection, system & technology, users, and usage. [Tsakonas et al. 2004] developed a framework to evaluate the interaction between the user and the DL. [Fuhr et al. 2007] provides a set of flexible and adaptable guidelines for DL evaluation, outlining the main directions, methods, and techniques for assessing the components of a DL. Approaches for DL Quality Evaluation > Broad Studies and Conceptual Frameworks

7. [Shen et al. 2006] proposes a model of DL success from the end user perspective, based on the integration of various research studies of different areas. [Kyrillidou and Giersch 2005] develops DigiQual, a protocol allowing users to answer about 12 quality themes to identify the best practices for a DL system. [Gonçalves 2003] and [Klass et al. 2006] develop and present standards for log formats aimed at registering data for DL evaluation. Approaches for DL Quality Evaluation > Broad Studies and Conceptual Frameworks

8. [Gonçalves et al. 2007] defines an explicit formal and quantitative quality model for digital libraries based on the 5S formal framework for digital libraries. [Moreira et al. 2007] describes a tool implementing a portion of the 5S quality model. Approaches for DL Quality Evaluation > Broad Studies and Conceptual Frameworks

9. Delos Reference Model: considers 3 levels, hierarchically related: 1)DL: where the digital contents are kept; 2)DLS: where is all the organizational and software application components that are able to manage the contents, providing useful services to the users; 3)DLMS: supports the DLS. aims at providing a representation which characterizes existing and future DLMS considering at least four perspectives: DL end- users, designers, system administrators, and application developers. Approaches for DL Quality Evaluation > The DELOS Approach

10. DL DLSDLMS

11. Approaches for DL Quality Evaluation > The DELOS Approach The entry point for all the concepts related to the content that is managed and disseminated by the DL, e.g., collections, information space model, metadata, ontologies;

12. Approaches for DL Quality Evaluation > The DELOS Approach The root for concepts like roles, communities, and profiles, that represent aspects of the DL users;

13. Approaches for DL Quality Evaluation > The DELOS Approach The entrance to that part of the model which concerns DL functions;

14. Approaches for DL Quality Evaluation > The DELOS Approach Regards software components, hosting nodes and how these are linked and constrained;

15. Approaches for DL Quality Evaluation > The DELOS Approach Groups qualitative parameters characterizing the digital library behavior within a given operational domain;

16. Approaches for DL Quality Evaluation > The DELOS Approach Covers all the concepts that are related to established procedures or plans of actions governing the DL, such as collection management, preservation, and access rights.

17. Approaches for DL Quality Evaluation > Notion of Quality in the DELOS Reference Model Quality is expressed by a set of quality parameters that can be measured; the measurements are mostly related to the DL content and functionalities. Quality can be considered at the three levels of abstraction: DL, DLS, and DLMS. – We can define quality parameters for the information objects, for the services, and for the system that supports the management of the services.

18. Approaches for DL Quality Evaluation > Notion of Quality in the DELOS Reference Model DL: the quality of information objects needs to be verified over acquisition and lifetime. Possible quality parameters: completeness, authenticity; DLS: inherits quality related concepts from the DL. DLMS:the DLMS is a software system with diverse components. The DL system administrator need to refer to software engineering best practices regarding software quality measurement.

19. Steps towards a Reference Quality Model for DLs 1.Contribute to the definition of a Reference Model for DL; 2.Formalization of the Model; 3.Definition and Formalization of Quality Indicators 4.Defining the context for each quality dimension in light of the Information Life Cycle 5.Discussion with the community and reformulation 6.Providing Support for the Model

20. Steps towards a Reference Quality Model for DLs 1.Contribute to the definition of a Reference Model for DL 2.Formalization of the Model; 5S Model [Goncalves et al, ACM TOIS, 2004] 3.Definition and Formalization of Quality Indicators 5S Quality Model [Goncalves et al, IP&M, 2007]

21. Steps towards a Reference Quality Model for DLs 4. Defining the context for each quality dimension in light of the Information Life Cycle

22. Steps towards a Reference Quality Model for DLs 1.Discussion with the community and reformulation 1.With Librarians and interested parties in [Goncalves 2007]; [Partially done here] 2.Providing Support for the Model 5SQual Tool [Moreira et al, JCDL, ECDL, 2007]

23. Conclusions A co-operative work to make some steps towards the definition of a complete Reference Quality Model for DL has been initiated having in mind the objective of defining, and developing, a model where all previous relevant experiences come together in a synergistic way

24. Acknowledgments This work was partially supported by the DELOS Network of Excellence on Digital Libraries, as part of the Information Society Technologies (IST) Program of the European Commission (Contract G038-507618). Support also was provided through the US National Science Foundation through grants IIS-0535057, DUE- 0532825, DUE-0435059, and IIS-0325579. and through the CNPq grant and CNPq individual grant to Marcos Andre Goncalves.