Benchmarking in North American Libraries: An Evolving Art International Conference on Performance Measurement and Benchmarking in Libraries Bratislava, Slovakia September 23, 2008 Duane Webster Executive Director Emeritus Association of Research Libraries ARL

Duane Webster, Executive Director Emeritus Thinking Strategically About Library Futures What is the central work of the library? How can we do more, differently be the most needed by our community of users in the next decade? and at less cost? What important set of services does the library provide that others can’t? What new roles are needed? How is user behavior changing? What should our libraries aspire to be ten years from now? What are the implications of technology driven change? What are the essential factors responsible for the success of the library?

What is Benchmarking ? Benchmarking is a process where libraries measure their performance and compare themselves to similar organizations. Benchmarking can focus on various aspects of libraries – size of investment, nature of collections and staff, library operations, or services provided A contemporary view of benchmarking is to focus on user perspectives of library performance Duane Webster, Executive Director Emeritus

How Does Benchmarking Work? Viewing libraries as organic systems: –Inputs –Processes and activities –Outputs (services and deliverables) –Impact (e.g. user perceptions of library performance) Using a variety of tools to collect information and measure success in each area Duane Webster, Executive Director Emeritus

Why is Benchmarking Important Today? Help libraries engage a changing environment and to do better of providing services Express the value of the library to its community of support Demonstrate stewardship and accountability for resources invested in library Duane Webster, Executive Director Emeritus

Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication. Members: 123 major research libraries in North America. Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources. Users: Three million students and faculty served. Expenditures: $3.4 billion annually, $1.1 billion for acquisitions of which 31 percent is invested in access to electronic resources.

Duane Webster, Executive Director Emeritus ARL Statistics and Measurement

. StatsQUAL ™

LibQUAL+™ Description LibQUAL+ TM is a research and development project undertaken to define and measure library service quality across institutions and to create useful quality- assessment tools for local planning.

PERCEPTIONS SERVICE “….only customers judge quality; all other judgments are essentially irrelevant” Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press. The LibQUAL+ ® Premise

Dimensions of Library Service Quality Information Control Library Service Quality Self-Reliance Equipment Timeliness Ease of Navigation Convenience Scope of Content Affect of Service Library as Place Reliability Assurance Responsiveness Empathy Refuge Symbol Utilitarian Space

Affect of Service §Assessing the attitudes and abilities of employees §“I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.” - Faculty member

Library as Place §Evaluating the library environment and its facilities §“I enjoy the libraries a lot and think they’re a great, quiet place to go to get away from everything and concentrate on work. They really let students work harder and have a comfortable place to study.” - UT Austin Undergraduate

Personal Control §Can library users navigate the library and its resources on their own terms? I want what I want, when I want it, in the format of my choice. §“By habit, I usually try to be self-sufficient. And I’ve found that I’m actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.” – Graduate student

Understanding LibQUAL+ ® Results §For the 22 items LibQUAL+ asks users’ to rate their: Minimum service level Desired service level Perceived service performance §This gives us a ‘Zone of Tolerance’ for each question; the distance between minimally acceptable and desired service ratings §Perception ratings ideally fall within the Zone of Tolerance

Using LibQUAL+ ® Results §Strategic Service Developments Data to support service development Ability to identify where not meeting expectations Measure if change has met need §Budget Discussions Data to support bid for increased funding Data to support case for change in emphasis (towards e-provision) §Marketing Position Status of the library within the University Importance of national & international benchmarking

LibQUAL+ ® Participants * 2007 data reflects Session I data only

Rapid Growth in Other Areas §Languages American English British English French Dutch Swedish §In development Chinese Greek Spanish German §Consortia Each may create 5 local questions to add to their survey §Types of Institutions Academic Health Sciences Academic Law Academic Military College or University Community College European Business Hospital Public State §Countries U.S., U.K., Canada, the Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia

Participating Libraries World LibQUAL+™ Survey

LibQUAL+ ® Languages Over 1000 institutions 1,000,000 respondents

In Closing LibQUAL+ ® §Focuses on success from the users’ point of view (outcomes) §Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense §Requires limited local survey expertise and resources §Analysis available at local, national and inter-institutional levels §Offers opportunities for highlighting and improving your status within the institution §Can help in securing funding for the Library

LibQUAL+ ® Resources §LibQUAL+ ® Website: §Publications: §Events and Training: §LibQUAL+ ® Bibliography: §LibQUAL+ ® Procedures Manual:

Duane Webster, Executive Director Emeritus The Challenge: Building Organizational Capacity Complementing LibQUAL+ ® with additional measures. Developing impact studies on user success, economic value, and community return on investment. Moving target: what is a digital library? E-Resources: understanding usage. Gaining acceptance and use of standard measures for e-resources. Building a climate of assessment throughout library.

Duane Webster, ARL Executive Director Emeritus In Closing Libraries need to demonstrate accountability and effectiveness. A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work. Basic questions of role, vision, and impact must be answered by library community.

Duane Webster, Executive Director Selected References Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, [Available at Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, [Available at Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services." ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): [Available at Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, [Available at Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, [Available at Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, [Available at Selected References