ARL Statistics and Measurement Program Effective, Sustainable and Practical Assessment Martha Kyrillidou Director, ARL Statistics and Service Quality Programs Association of Research Libraries Steve Hiller Director, Assessment & Planning, University of Washington Libraries ARL Visiting Program Officer Jim Self Director, Management Information Services, University of Virginia Library ARL Visiting Program Officer University of Cape Town Libraries 16 August 2007 ARL

ARL Mission Non-profit organization of the libraries of research institutions in North America Forum for exchange of ideas Agent for collective action

Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication. Members: 123 major research libraries in North America. Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources. Users: Three million students and faculty served. Expenditures: $3.4 billion annually, $1.1 billion for acquisitions of which 37 percent is invested in access to electronic resources.

What makes a research Library? Breadth and quality of collections and services Sustained institutional commitment to the library Distinctive resources in a variety of media Services to the scholarly community Preservation of research resources Contributions of staff to the profession Effective and innovative use of technology Engagement of the library in academic planning -- from ARL ‘Principles of Membership’

The Beauty and the Beast What makes a quality research library? “Quality much like beauty is in the eye of the beholder”

The Beauty and the Beast What makes a quality research library? “Quality much like beauty is in the eye of the beholder”

The Beauty and the Beast What makes a quality research library? “Quality much like beauty is in the eye of the beholder”

A gateway to assessment tools:ARL StatsQUAL ™: ARL Statistics -- E-Metrics LibQUAL+ ® DigiQUAL ™ MINES for Libraries ™ Library Assessment Conferences Service Quality Evaluation Academy Library Assessment blog Making Library Assessment Work ESP Assessment –Effective, Sustainable, Practical Assessment at ARL

. StatsQUAL ™ Duane Webster, Executive Director

StatsQUAL ™

Updating the Traditional ARL Statistics E-Metrics = ARL Supplementary Statistics –On going efforts to update and refine core data. –Exploring feasibility of collecting e-metrics. ARL Task Force on New Ways of Measuring Collections : –Growing concern with utility of membership index. –Study ARL statistics to determine relevance. –Develop Profile of Emerging Research Libraries.

Tradition ARL membership criteria index: –Volumes held –Volumes Added Gross –Current Serials –Total Expenditures –Professional plus support staff

Task Force Recommendations Reserve use of the current membership criteria index to those occasions when it is needed for consideration of membership issues Implement an expenditure-focused index

Use the new expenditure-focused index for any public reports, such as the Chronicle of Higher Education Begin to develop a services-based index that combines the following three factors: collections, services, and collaborative relationships

Revise definitions for collections-related data categories, such as serials, and experiment with a variety of new measures, including usage data, strength of collections, and service quality measures to develop a richer set of variables for potential inclusion in the three factor alternative index

Collect qualitative data to develop a profile of ARL member libraries The power of narrative … (Peter Brophy, Yvonna Lincoln, Colleen Cook …..)

The focus on economics ARL Expenditures focused index: –Total library expenditures –Expenditures for Professional staff –Expenditures for Library materials –Professional plus support staff

Thinking Strategically About Library Futures What is the central work of the library and how can we do more, differently, and at less cost? What important set of services does the library provide that others can’t? What new roles are needed? What advantages does the research library possess? What will be the most needed by our community of users in the next decade? How is user behavior changing? What should our libraries aspire to be ten years from now? What are the implications of technology driven change? What are the essential factors responsible for the success of the library?

Defining Success in a Digital Environment Crafting new measures of success. Moving from measuring inputs to outputs. Understanding impact of library roles and services. Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals. Reallocating and managing capabilities to focus on new definitions of success.

ultimately "the ideal" is one big digital library of books held in libraries around the world."You're going to be able to go back and forth from Google Book Search into Harvard, or from Harvard's catalog into Google,“ reminding users that it was important to keep the HOLLIS catalog the "starting point for their research," since only HOLLIS can give a full picture of the 15.5 million books in Harvard's collection, the nation's largest …Harvard would look to scan all of its-out-of copyright books, about one million, over time but that ultimately "the ideal" is one big digital library of books held in libraries around the world. ”It shouldn't matter whether it came from Harvard or Michigan"HOLLIS catalog -Sidney Verba, Harvard library director, -Library Journal Academic Newswire – April 24, 2007

old.libqual.org A LibQUAL+ ® Update §The LibQUAL+ ® premise, dimensions, and methodology §LibQUAL+ ® results §LibQUAL+ ® in action

old.libqual.org Dimensions of Library Service Quality Information Control Library Service Quality Model 3 Self-Reliance Equipment Timeliness Ease of Navigation Convenience Scope of Content Affect of Service Library as Place Reliability Assurance Responsiveness Empathy Refuge Symbol Utilitarian Space

old.libqual.org Survey Instrument – “22 items…

old.libqual.org …and a Box” §Why the Box is so Important: About 40% of participants provide open- ended comments, and these are linked to demographics and quantitative data Users elaborate the details of their concerns Users feel the need to be constructive in their criticisms, and offer specific suggestions for action

Background: ServQUAL  LibQUAL+ ®  DigiQUAL ™ LibQUAL+ ® Dimensions of Service Quality: Affect of Service Information Control Library as Place Developing DigiQUAL ™ Survey Items DigiQUAL™ 12 themes of service quality: Accessibility Navigability Interoperability Collection building Resource Use Evaluating collections DL as community for users DL as community for developers DL as community for reviewers Copyright Role of Federations DL Sustainability

Assessing the Value of Networked Electronic Services: Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™ The MINES survey

A research methodology consisting of a web-based survey form and a sampling plan. Measures who is using electronic resources, where users are located at the time of use, and their purpose of use. Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E- Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+ ®, DigiQUAL ™ ). What is MINES?

How extensively do sponsored researchers use the new digital information environment? Are researchers more likely to use networked electronic resources from inside or outside the library? Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on- campus, but not in the library; or off-campus)? What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys? Are particular network configurations more conducive to studies of digital libraries patron use? Questions Addressed

Where are the most critical assessment needs and opportunities? Complementing LibQUAL+ ® with additional measures. Developing impact studies on user success, economic value, and community return on investment. Moving target: what is a digital library? E-Resources: understanding usage. Gaining acceptance and use of standard measures for e-resources. Building a climate of assessment throughout library.

What are the lessons learned? Understanding changes in users approach to information resources. Service quality improvement is a key factor. Understanding the impact of e-resources on library services - TRL. Learning how to compete and/or collaborate with Google. Upfront investment in design and development. Making the assessment service affordable, practical, & effective. Assessment needs to be satisfying and fun.

In Closing As higher education is challenged on accountability and effectiveness issues so will libraries. A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work. Basic questions of role, vision, and impact must be answered by library community.

What’s in a “Library” A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. –--Justice Oliver Wendell Holmes

Selected References Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, [Available at Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, [Available at Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services." ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): [Available at Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, [Available at Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, [Available at Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, [Available at Selected References