1. Evaluation of digital Libraries: Criteria and problems from users’ perspectives Article by Hong (Iris) Xie Discussion by Pam Pagels

2. The Digital library as special case Definitions of digital library include: Access and retrieval of digital content Collection, organization, and service aspects of digital resources

3. Association of Research Libraries ARL states common elements of digital libraries definitions DLs are not a single entity They require technology to link resources Linkages between DLs and information services are transparent to end users Universal access is a goal DL collections are not limited to document surrogates

4. Issues of DL evaluation and criteria Research is “still in its infancy.” Who should evaluate? What to evaluate? How to evaluate? Why to evaluate?

5. Don’t forget the users! Xie states, “users of DLs should have their voices heard.” Marchionini, Plaisant, and Komlodi (1998) state, “…all efforts to design, implement, and evaluate DLs must be rooted in the information needs, characteristics, and contexts of the people who may use those libraries.”

6. Questions to consider What criteria do users identify as important for the evaluation of DLs? What are the problems with existing DLs?

7. Evaluation “…a judgment of worth.” (Xie) “a research process that aims to understand the meaning of some phenomenon situated in a context and changes that take place as the phenomenon and the context interact.” (Marchionini)

8. DL evaluation research Usability attributes Learnability Efficiency Memorability Errors Satisfaction

9. Studies also address Content (scope and accuracy) System performance (interface, electronic publishing) Learning and teaching of a system Impact on users and their communities; acceptance

10. Methods for obtaining data Focus groups Interviews Observations Usability testing User registration and transaction logging User surveys

11. Xie’s study Objectives for the study participants: Develop criteria for evaluation of DLs Evaluate existing DLs by applying evaluation criteria developed by themselves

12. Methodology 48 subjects recruited from the School of Information Studies, Univ. Wisconsin-Milwaukee; data collected twice, in 2003 and 2004 Before data collection, subjects were given ARL definitions of DLs Data were collected mainly from an open-ended self- administered survey; one week response time

13. Data Analysis Quantitative and qualitative methods Quantitative methods included descriptive data analysis such as frequency and mean Qualitative methods included content analysis and developing categories and subcategories of evaluation criteria and problems Criteria and types of problems were derived directly from users instead of existing structures

14. Xie’s results Usability was most important criterion for a useful digital library Interface design Search and browse features Navigation efficiency that is intuitive Help features readily identified and available View and output

15. Also important Collection quality, criterion of traditional libraries Quality of content, what distinguishes the DL from content on the free Web Subject, scope, coverage, and authority Currency of content and updates Service quality DLs mission and intended audience Unique services

16. Also, also important System performance Like information retrieval (IR) systems, DLs require efficiency of information retrieval—how quickly the user can find needed information User opinion solicitation User feedback Ability to learn from its users Contact information

17. Problems with existing DLs Usability issues Interface design: search features, navigation, help features Output options Collection quality/quality control Coverage of the collection was considered good, but information regarding authority, accuracy, currency, and copyrights was difficult to find

18. Problems, continued Service quality Lack of a community service Not dynamic enough System performance Needs greater precision Response time Contact information needed for feedback

19. Discussion questions User criteria vs. designer/administrative criteria: Is one set more important than the other? Does this study offer new insight for DL evaluation criteria or continue/extend extant traditional library methodology? What are the strengths of the study? Weaknesses? What criteria/methodology/data might we add to this study to improve it?