Presentation on theme: "Performance Measurement in Academic Libraries Martha Kyrillidou, ARL Steve Hiller, University of Washington Jim Self, University of Virginia EBLIP 4 Workshop."— Presentation transcript:
Performance Measurement in Academic Libraries Martha Kyrillidou, ARL Steve Hiller, University of Washington Jim Self, University of Virginia EBLIP 4 Workshop North Carolina May 11, 2007
ARL Tools and R&D Martha Kyrillidou Director, ARL Statistics and Service Quality Programs Association of Research Libraries
ARL “Life is not measured by the breaths we take...but by the moments that take our breath away.”
ARL chart a course for every endeavor that we take the people's money for, see how well we are progressing, tell the public how we are doing, stop the things that don't work, and never stop improving the things that we think are worth investing in. –President William J. Clinton, on signing the Government Performance and Results Act of 1993
ARL University of Hawaii Goal 1: Educational Effectiveness and Student Success Goal 2: A Learning, Research, and Service Network Goal 3: A Model Local, Regional, and Global University Goal 4: Investment in Faculty, Staff, Students, and Their Environment Goal 5: Resources and Stewardship »Measuring Our Progress Report 2006, #satisfaction #satisfaction
ARL University of Hawaii (cont’d) Student Engagement How engaged are University of Hawaii students in their educational experience at upper division/four- year campuses? Benchmark #1 Level of Academic Challenge Benchmark #2 Active and Collaborative Learning Benchmark #3 Student-Faculty Interaction Benchmark #4 Enriching Educational Experiences Benchmark #5 Supportive Campus Environment
ARL University of Hawaii (cont’d) Information and Technology Resources: Library How does U H’s major library compare on a national basis? U H Manoa ranks 68th among the 113 ranked university libraries that are members of the Association of Research Libraries (A R L). Source: 2003–2004 A R L Membership and Statistics The indexed ranking is based on the number of volumes held, number of volumes added in the last fiscal year, number of current serials, number of permanent staff, and total operating expenditures. The library aspires to regain its previous higher standing which was significantly impacted by budget cuts in the mid- to late 1990s, and from which the library has been slowly recovering.
ARL Mission:Shaping the future of research libraries in the changing environment of public policy and scholarly communication. Members: 123 major research libraries in North America. Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources. Users: Three million students and faculty served. Expenditures: 37 percent is invested in access to electronic resources.
ARL Library Assessment in an Electronic Era What are some of the current developments with library assessments efforts? ARL StatsQUAL ™ E-Metrics LibQUAL+ ® DigiQUAL ™ MINES for Libraries ™ Where are the most critical needs and opportunities? What are the lessons learned?
ARL Thinking Strategically About Libraries’ Futures What is the central work of the library and how can we do more, differently, and at less cost? What important set of services does the library provide that others can’t? What new roles are needed? What advantages does the library possess? What will be the most needed by our community of users in the next decade? How is user behavior changing? What should our libraries aspire to be ten years from now? What are the implications of technology driven change? What are the essential factors responsible for the success of the library?
ARL Defining Success in a Digital Environment Crafting new measures of success. Moving from measuring inputs to outputs and outcomes Understanding impact of library roles and services. Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals. Reallocating and managing capabilities to focus on new definitions of success.
ARL Updating the Traditional ARL Statistics E-Metrics = ARL Supplementary Statistics –On going efforts to update and refine core data. –Exploring feasibility of collecting e-metrics. ARL Task Force on New Ways of Measuring Collections : –Growing concern with utility of membership index. –Study ARL statistics to determine relevance. –Develop Profile of Emerging Research Libraries.
ARL The LibQUAL+ ® Update The LibQUAL+ ® premise, dimensions, and methodology LibQUAL+ ® Results LibQUAL+ ® in Action
The LibQUAL+ ® premise, dimensions, and methodology
ARL The Need for LibQUAL+ ® Underlying need to demonstrate our worth The reallocation of resources from traditional services and functions Rapid shifts in information-seeking behavior Need to keep abreast of customer demands Increasing user demands
ARL Why Use LibQUAL+ ® ? Feedback from LibQUAL+ ® Users “Why did you choose to use LibQUAL+ ® ?” LibQUAL+ ® was recommended to us as offering a well designed, thoroughly Library-focused set of survey tools Opportunity to benchmark Cost-effectiveness Automated processing & fast delivery of results Respectability and comparability (with others and historically)
ARL The LibQUAL+ ® Premise PERCEPTIONS SERVICE “….only customers judge quality; all other judgments are essentially irrelevant” Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.
ARL Multiple Methods for Listening to Customers Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture * A SERVQUAL-type instrument is most suitable for these methods
ARL World LibQUAL+ ® Survey 2005
ARL LibQUAL+ ® Languages Over 1,000 institutions 1,000,000 respondents
ARL Survey Instrument – “22 items…
ARL and A Box.” Why the Box is so Important: –About 40% of participants provide open-ended comments, and these are linked to demographics and quantitative data –Users elaborate the details of their concerns –Users feel the need to be constructive in their criticisms, and offer specific suggestions for action
LibQUAL+ ® Results
ARL Understanding LibQUAL+ ® Results For the 22 items LibQUAL+ asks users’ to rate their: Minimum service level Desired service level Perceived service performance This gives us a ‘Zone of Tolerance’ for each question; the distance between minimally acceptable and desired service ratings Perception ratings ideally fall within the Zone of Tolerance
ARL General Findings Highly desired Making electronic resources accessible from my home or office Print and/or electronic journals I require for my work A haven for study, learning or research Lowest Library staff who instil confidence in users Giving users individual attention Space for group learning and group study
LibQUAL+ ® in Action
ARL Using LibQUAL+ ® Results Strategic Service Developments Data to support service development Ability to identify where not meeting expectations Measure if change has met need Budget Discussions Data to support bid for increased funding Data to support case for change in emphasis (towards e-provision) Marketing Position Status of the library within the University Importance of national & international benchmarking
ARL In Closing, LibQUAL+ ® : Focuses on the users’ point of view (outcomes) Requires limited local survey expertise and resources Analysis available at local, national and inter-institutional levels Offers opportunities for highlighting and improving your status within the institution Can help in securing funding for the Library
ARL "Each organization must create and communicate performance measures that reflect its unique strategy." Dr. Robert S. Kaplan, Harvard Business School
Developing the DigiQUAL ™ Protocol for Digital Library Evaluation Building on the LibQUAL+ ® experience Secures feedback on user’s perceptions of library’s web site Five questions on services, functionality, and content Goal is to determine utility, reliability, and trustworthiness
ARL DigiQUAL™ Dimensions Accessibility Navigability Interoperability Collection building Resource Use Evaluating collections DL as community: users, developers, reviewers Copyright Role of Federations DL Sustainability
ARL Outstanding Issues and Challenges Unique DLs: niche market, critical mass, both? Balance: –custom vs. generic content results –flexible vs. standard implementation scaling Mixed methods –Preserving user privacy –Collecting truly useful data Moving target: digital libraries as… it depends.
ARL Assessing the Value of Networked Electronic Services The MINES Survey Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™
ARL What is MINES for Libraries™? A research methodology consisting of a web- based survey form and a sampling plan. Measures who is using electronic resources, where users are located at the time of use, and their purpose of use. Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, Different from other electronic resource usage measures that quantify total usage or measure how well a library makes electronic resources available.
ARL Questions Addressed How extensively do sponsored researchers use the new digital information environment? Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off- campus)? What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys? Are particular network configurations more conducive to studies of digital libraries patron use?
ARL Library User Survey
ARL What are the most critical assessment needs and opportunities? Complementing LibQUAL+ ® with additional measures. Developing impact studies on user success, economic value, and community return on investment. Moving target: what is a digital library? E-Resources: understanding usage. Gaining acceptance and use of standard measures for e- resources. Building a climate of assessment throughout library.
ARL What is the lesson learned? Building standardized assessment methods and tools are a key component of understanding users, performance measurement, and improvement of services.
ARL In Closing As higher education is challenged on accountability and effectiveness issues so will libraries. A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work. Basic questions of role, vision, and impact must be answered by library community.
ARL What’s in a word? LIBRARY
ARL A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used. --Justice Oliver Wendell Holmes
User Needs Assessment or Through the Looking Glass Steve Hiller Director of Assessment and Planning University of Washington Libraries
ARL How have things changed as far as getting your information in the past 5 years ? “We never have to go to the library.” (sounds of laughter and lots of paper ripping noise on audio tape) Faculty Focus Group 2000 (UW College of Education) User Needs Assessment or Through the Looking Glass
ARL Traditional Library Core Business Physical Collections –Print (primarily) –Microforms –Other (minor) Facilities –House collections –Customer service and work space –Staff work space Services –Reference –Instruction –Access
ARL Use of UW Physical Collections To DOWN
ARL UW Libraries Gate Counts DOWN
ARL UW Libraries In-Person Reference Queries DOWN
ARL Time for a New Business Model? Try the Customer-Centered Library All services & activities viewed through the eyes of customers Customers determine quality Library services and resources add value to the customer Not better libraries... Not better customers... but Assess/Measure the Value the Library Provides the University Community “Documenting the libraries contributions to quality teaching, student outcomes, and research productivity will become critical.” (Yvonna Lincoln 2006)
ARL The Value of User Needs Assessment Decisions based on data not assumptions - “assumicide” Fundamental to User-Centered Library Users determine quality, importance and success Evaluation and assessment focus on user outcomes Align collections, resources and services with user needs Identify differences/similarities in needs and use by academic areas/groups Data that can help tell our story Ensure libraries are responsive to their communities
ARL What We Need to Know to Support Our Communities Who are our customers (and potential customers)? What are their teaching, learning, clinical and research interests? How do they work? What’s important for their work? What are their library and information needs? How do they currently use library/information services? How would they prefer to do so? How do they differ from each other in library use/needs? How does the library add value to their work?
ARL Understand Differences in Your Community Library and information needs and use may differ substantially by academic area, groups, and culture Identifying and understanding these differences enables libraries to target and market services that add the most value for each group or area Multiple assessment methods, including both quantitative and qualitative data, can identify differences and provide the most comprehensive picture of these communities
ARL Multidimensional Library Assessment: Beyond Counts and Satisfaction Surveys Data based decision making needs good data source Use multiple assessment methods Focus on user work and their information seeking and using behavior Increased reliance on qualitative data to identify issues from the perspective of users Learning from our users Partnering with other campus programs Repurposing existing data when possible
How Do We Measure Value?
ARL UW Assessment Priorities Customer Needs, Use and Success Information seeking behavior and use Patterns of library use Value of library User needs Library contribution to customer success User satisfaction with services, collections, overall Data to make informed and wise decisions that lead to resources and services that contribute to user success
ARL University of Washington Libraries Assessment Methods Used Large scale user surveys every 3 years (“triennial survey”): 1992, 1995, 1998, 2001, 2004, 2007 –All faculty –Samples of undergraduate and graduate students –Research scientists, Health Sciences fellow/residents In-library use surveys every 3 years beginning 1993 LibQUAL+™ from Focus groups/Interviews (annually since 1998) Observation (guided and non-obtrusive) Usability Use statistics/data mining Information about assessment program available at:
ARL Our Latest Assessment Method
ARL UW Triennial Library Use Survey Number of Respondents and Response Rates Large number of respondents allows for analysis within groups Faculty % % % % % Grad Student % % % % % Undergrad502 25% % % % %
ARL Reasons for In-Person Library Visits 2001 Faculty and Undergrads Visiting Weekly or More Often
ARL Changes in UW Libraries Use Patterns (% of each group who use library at least weekly) Visit 1998 Visit 2001 Visit 2004 Remote 1998 Remote 2001 Remote 2004 Faculty Change 47%40% -15% 29% -28% 73%79% +8% 91% +15% Grad Change 78%59% -24% 52% -12% 63%75% +19% 87% +16% Undergrad Change “Visit library for” 67% (78%) 61% -9% (72%) 61% (76%) 43%54% +26% 57% +6%
ARL E-Journals Drive Remote Use Look for E-Journals at Least 2x week Faculty by Area Journal Article Downloads 4,761,704 ( )
ARL Resource Type Importance Faculty By Selected Colleges
ARL Overall Collections Satisfaction By Group in Selected Colleges
ARL Overall Satisfaction by Group
ARL Top Priorities by UW Group
ARL UW Faculty Top Priorities by Academic Area (2004)
ARL UW Libraries 2007 Triennial Survey – Outcomes Question for Faculty 9. What contribution does the UW Libraries make to: (Likert scale of 1 (minor) to 5 (major) Keeping current in your field Finding information in related fields or new areas Being a more productive researcher Being a more effective instructor Enriching student learning experiences Helping you make more efficient use of your time Recruiting colleagues and students to UW
ARL Use of Physical Library 2005 In-Library Use Survey (Triennial) One page survey distributed to people entering 15 campus libraries during 4-6 two hour time blocks 3861 surveys returned (minimum of 50 for any library) Surveys asked: –What did you do in this library today* –How often do you use this library –How important are these services to you* –How would you rate these library services and resources* –What can we do to make this library better for you (write in) *Option for local library question
ARL Respondents by User Group for Selected Libraries
ARL The Qualitative Provides the Key Increasing use of such qualitative methods as, comments interviews, focus groups, usability, observation Statistics often can’t tell us –Who, how, why –Value, impact, outcomes Qualitative provides information directly from users –Their language –Their issues –Their work Qualitative provides understanding
ARL Information Literacy Focus Groups: Findings The information environment is too complex General search engines (e.g. Google) are preferred over library licensed/provided interfaces Undergrads have difficulty determining which library sources to use Faculty “dumbing down” library research assignments Ubiquity of library research – any place, any time has changed research patterns Availability online is more efficient way to research The personal connection with a librarian is important
ARL Guided Observation 2003 Bibliographic Database Searching Faculty and graduate students search very differently than we think they should Common observations included: –Prefer to use single keyword search box –Little use of Boolean commands –Limits or format changes rarely employed –Commands need to be on first page or lost –Visible links to full-text critical Important features for librarians are not necessarily important to faculty and students
ARL Understanding How Researchers Work: Four Recent Studies Focus first on work of faculty/grad students and then on connection to library University of Minnesota Humanities/Social Science research (2005 Mellon funded) New York University 21 st Century Library Study (2006) Ithaka Faculty survey and interviews on scholarly communication (2006) University of Washington Biosciences Review (2006)
ARL Initial Results University of Minnesota –Extremely comfortable with electronic sources –Inadequate methods for organizing their research materials New York University –Researchers (all disciplines) no longer tied to physical library –Expectations for info shaped by Web and commercial sector Ithaka –Prefer publishing in widely circulated journals at no cost to them –Biologists frequently search for information outside their area University of Washington –Start info search outside library space (virtual and physical) –Could not come up with “new library services” unprompted
ARL Biosciences Review Task Force ( ): Reasons for Review Better understand how bioscientists work Growing interdisciplinarity Significant change in use patterns Libraries organizational structure in subject- based silos Value of research enterprise to the University Strengthening library connection to research
ARL Biosciences Review Process Define scope Mine existing data Acquire new information –Environmental scan Jan-May 2006 –Interviews (biosciences faculty) Feb 2006 –Focus groups (biosci faculty & students) Mar-Apr 2006 –Peer library surveys Apr 2006 Synthesis and first draft May-Aug 2006 Final report and recommendations Sep-Dec 2006 Incorporate into Libraries plan 2007-
ARL UW Students, Faculty and Doctorates Awarded by Academic Area
ARL Physical Library Use by Group and Academic Area (2005 In-Library Use Survey)
ARL Top 4 Faculty Priorities by Academic Area
ARL Faculty Interview Themes Library seen primarily as E-Journal provider Physical library used only for items not available online Start information search with Google and PubMed Too busy for training, instruction, workshops Faculty who teach undergrads use libraries differently Could not come up with “new library services” unprompted
ARL Focus Group Themes Print Is Dead, Really Dead; Our Virtual Space Not Yours Google, PubMed, Web of Science starting points for all Faculty identify library with E-journals Want more online, including older materials; if not online deliver digitally Faculty/many grads go to physical library as last resort Too many physical libraries E-Science emerging as a new priority Lack understanding of many library services, resources Undergrads rarely use print unless assigned by faculty Increasing overlap between “bio” research and other science research
ARL Task Force Recommendations Consolidate collections and service points –Reduce print holdings; focus on online resources Reorganize libraries around broad user communities Integrate search/discovery tools into users workflow Expand/improve information/service delivery options Use an integrated approach to collection allocations Increase integration of librarians with user workflow Lead scholarly communication and e-science work Partner with broader science/technology community Provide more targeted communication and marketing
ARL What We’ve Learned about the UW Community Libraries are still important source of information used for teaching, learning and research but lessening in value Library needs/use patterns vary by and within academic areas and groups Remote access is preferred method and has changed the way faculty and students work and use libraries Faculty and students find information and use libraries differently than librarians prefer them too Library/information environment is perceived as too complex; users find simpler ways (Google) to get info Customers cannot predict the Libraries future
From Data to Outcomes Jim Self Director, Management Information Services University of Virginia Library
ARL The next topics Using data for improvement and reassurance –LibQUAL+ as a case study –Corroboration –Benchmarking The Balanced Scorecard
“…but to suppose that the facts, once established in all their fullness, will ‘speak for themselves’ is an illusion.” Carl Becker Annual Address of the President of the American Historical Association, 1931
LibQUAL+ ® Finding the right numbers
ARL Initial Examination Focus on the separate user categories –Faculty –Undergraduates –Graduate students Tally the number of responses –100+ needed in each category
ARL UVa 2006 Responses per category –Faculty –Undergraduates –Graduate students
ARL The 22 core questions Examine the notebooks –Scan results for each user category –Scan the summary charts by dimension –Rearrange the dimension charts Create thermometer graphs –Question by question –22 bars for each user category Identify the red zones
ARL Red Zones at UVa Journal Collections –Faculty –Grad Students Website –Faculty Remote access –Grad students
ARL Following up with Journals Who is unhappy? –Drill down by discipline and user category. Why are they unhappy? –Read the comments –Conduct targeted interviews
ARL Among the 22 core questions, the most desired are: Faculty –Journal collections 8.60 –Web site 8.49 Grad Students Journal collections 8.61 Remote access 8.53 Undergrads –Modern equipment 8.35 –Comfortable and inviting location 8.29 –Space that inspires study and learning 8.29
ARL Among the 22 core questions, the least desired are: Faculty –Community space for group learning 6.56 –Quiet space for individual activities 7.03 Grad Students Community space for group learning 6.86 Giving users individual attention 7.27 Undergrads –Giving users individual attention 7.06 –Employees who instill confidence 7.30
ARL Benchmarking with Peers: General Satisfaction Questions In general, I am satisfied with the way I am treated at the library. In general, I am satisfied with library support for my learning, research, and/or teaching. How would you rate the overall quality of the service provided by the library?
ARL …beyond LibQUAL+ ® Corroboration Data mining Balanced Scorecard
ARL Corroboration Data are more credible if they are supported by other information John Le Carre’s two proofs
ARL Analyzing U.Va. Survey Results Two Scores for Resources, Services, Facilities –Satisfaction = Mean Rating (1 to 5) –Visibility = Percentage Answering the Question Permits comparison over time and among groups Identifies areas that need more attention
ARL Reference Activity and Visibility in Student Surveys
ARL Investment and Customer Activity University of Virginia Library
ARL Benchmarking Comparisons with Peers Within the University Within ARL
ARL Total Expenditures at UVA Other Academic Support (+200%) Research (+219%) Total Academic Division (+140%) Libraries (+81%) Instruction (+80%)
ARL Collections Expenditures UVA vs. ARL
ARL Median Faculty Salaries University of Virginia Library Compared to ARL Median
ARL The Balanced Scorecard A layered and categorized instrument that –Identifies the important statistics –Ensures a proper balance –Organizes multiple statistics into an intelligible framework
ARL The Balanced Scorecard Reflects the organization’s vision Clarifies and communicates the vision Provides a quick, but comprehensive, picture of the organization’s health
ARL The scorecard measures are “balanced” into four areas The user perspective The finance perspective The internal process perspective The learning and growth perspective
ARL Metrics Specific targets indicating full success, partial success, and failure At the end of the year we know if we have met our target for each metric The metric may be a complex measure encompassing several elements
ARL The BSC at the U.Va. Library Implemented in 2001 Reports for FY02 to FY06 Completing metrics for FY08 Will tally FY07 in July and August A work in progress
ARL Core Questions User Perspective –How well is the library meeting user needs? Internal Processes –Do the library’s processes function efficiently? Finance –How well are the library’s finances managed? Learning and Growth –Is the library well positioned for the future?
ARL Metric U.1.A: Overall rating in student and faculty surveys Target1: An average score of at least 4.00 (out of 5.00) from each of the major constituencies. Target2: A score of at least FY06 Result: Target1 –Graduate students 4.08 –Undergraduates 4.11
ARL Metric U.3.A: Circulation of new monographs Target1: 60% of newly cataloged monographs should circulate within two years. Target2: 50% of new monographs should circulate within two years. Result FY06: Target 1. –61.2% circulated (15,213 out of 24,852)
ARL Metric U.4.B: Turnaround time for user requests Target1: 75% of user requests for new books should be filled within 7 days. Target2: 50% of user requests for new books should be filled within 7 days. Result FY06: Target1. –79% filled within 7 days.
ARL Metric F.1.B.: Library spending compared to University expenditures Target1: : The University Library will account for at least 2.50% of the University’s academic division expenditures. Target2: : The Library will account for at least 2.25% of expenditures. Result FY06: Target1. –2.57% ($25.2M of $972M)
ARL Metric F.2.A: Unit Cost of Electronic Serial Use Target1: There should be no increase in unit cost each year. Target2: Less than 5% annual increase in unit cost. Result FY06: Target not met. –8.8% increase ($2.10 vs. $1.93)
ARL Metric I.1.A: Processing Time for Routine Acquisitions Target1: 90% of in-print books from North America should be processed within one month. Target2: 80% should be processed within one month. Result FY06: Target2. –87.2% processed.
ARL Metric I.2.A.: Internal Communications Metric I.2.A.: Internal Communications Target1: Positive scores (4 or 5) on internal communications statements from 80% of respondents in the biennial work-life survey. Target2: Positive scores from 60%. Result FY06: Target not met. –48% gave positive scores.
ARL Metric L.3.A.: Expenditures for digital materials Target1: Rank in the top 25% of ARL libraries in percentage of collections dollars spent on digital materials. Target2: Rank in top 33%. Result FY06: Target not met. –Ranked 74 of 109.
ARL Metric L.2.C.: Compare staff salaries to peer groups. Target1: Library faculty salaries should rank in the top 40% of salaries at ARL libraries. Target2: Rank in top 50%. Result FY06: Target1. –Ranked 33 of 113. (Top 28%)