Presented by Beverly Choltco-Devlin Reference and Electronic Resources Consultant Mid-York Library System September 25, 2009 REVVED UP FOR REFERENCE CONFERENCE Ithaca, NY

 What is Service Quality? ◦ Simply put, service quality is defined as “how well a service or activity is done”. Bertot, John Carlo, “Measuring Service Quality in the Networked Environment: Approaches and Considerations”, Library Trends, Spring 2001  What is Service Quality Assessment? ◦ Assessment or evaluation is the process through which we determine service quality



 Gap Model Hernon, P. based on A. Parasuraman, Valarie A. Zeithaml and Leonard L. Berry  Statistics, Performance Measures and Quality Standards C. McClure, R. D. Lankes, M. Gross, B. Choltco-Devlin

 Determining "the gap between customer expectations of those services of an academic or public library in general and those perceptions of the services offered by a particular academic or public library." Delivering Satisfaction and Service Quality: A Customer-Based Approach for Libraries, Peter Hernon and John R. Whitman. Chicago: American Library Association, 2001

 Using a variety of methodologies which can be customized and also applied to the service as a whole or specific components of the service. ◦ Statistics - Sets of raw numerical data ◦ Performance Measures - Measurements for evaluation purposes utilizing points of comparison using statistics or data from qualitative analysis ◦ Quality standards - A specific statement of the desired or expected level of performance that should be provided by a service or some aspect of that service McClure, Lankes, Gross, Choltco-Devlin



 Determine the degree to which service objectives are being met  How well do virtual reference services support larger organizational objectives  Produce trend data  Analyze costs vs. benefits within service-accountability  Compare cost/benefit in relation to other services

 Planning  Marketing/Promotion  Inform  Identify areas for improvement  Encourage thinking in terms of outcomes, results and most importantly impacts.

 Number of virtual reference queries received  Number of answers given  Total transactions  Number of questions received  Number of questions received virtually but not answered or responded to by completely virtual means (e.g., referrals, phone calls)  Type of virtual question received  Number of lost connections  Average transaction time  Number of repeat users  Number of abandoned transactions

 Percentage of virtual reference question per total reference questions  Sources used per question  Virtual reference question correct answer fill rate  Saturation rate (easier for academic libraries than public and very tricky in a virtual reference cooperative) ◦ The measure of the ration of digital reference service users who are members of a target population to total number of members of a target population

 Log and Report Analysis ◦ AskUs 24/7 and OCLC Questionpoint provide excellent methods for gathering data and generating reports ◦ Can look at such things as number of sessions, user’s browser, platform, service by time of day and/or day of week. ◦ If doing other types of virtual reference (i.e., , texting, im) these measures are also useful

 User satisfaction is closely related to but not exactly the same as service quality  These are often qualitative rather than quantitative ◦ Awareness of service ◦ Accessibility ◦ Expectations for service ◦ The transaction process ◦ Reasons for use ◦ Reasons for non-use ◦ Improvements needed ◦ Impact on user

 Raw cost of the service (subscription fees, software licensing)  Allied costs (personnel, tech support, etc.)  Cost as percentage of total reference budget  Cost as percentage of total organizational budget

 Training  Skill sets – what special skills are required to engage in a virtual reference transaction that are distinct from traditional face to face reference  Schedule allocation  Tech support

 Quality digital resources  Ease of accessibility  Authentication for premium resources  Licensing

 A quality standard is a specific statement of the desired or expected level of performance that should be provided by a service or some aspect of that service

 Users of the virtual reference service will rate the courtesy of the library staff as a score of X on a scale of 1 to 7 with 1 being discourteous to 7 being discourteous ◦ This quality standard will address the special issues arising from a virtual reference transaction where traditional non-text cues are absent and expectations regarding courtesy and how it is manifested can be radically different than that of a traditional reference transaction

 Requests for information on a patron’s pin number will only be referred to patron’s own library if information is not readily available on the patron’s library policy page in a consortium environment  Virtual reference transactions will have a 95% correct fill rate  90% of questions marked as follow-up will be responded to within 24 hours.  100% of reference questions will be responded to (but not necessarily completed within 24 hours of receipt

 Virtual reference service evaluation: Adherence to RUSA behavioral guidelines and IFLA digital reference guidelines ◦ ◦ This link is useful for information about the book itself but also the reference list at the bottom

 LIBQUAL+ and LIBQUAL-Lite (coming 2010) ◦ ◦ What is LibQUAL+®? LibQUAL+® is a suite of services that libraries use to solicit, understand, and act upon users’ opinions of service quality. These services are offered through the Association of Research Libraries (ARL) New Measures and Assessment Initiatives. The program’s centerpiece is a rigorously tested Web-based survey bundled with training that helps libraries assess and improve library services, change organizational culture, and market the library. LibQUAL+® enables systematic assessment and measurement of library service quality, over time and across institutions.

 EDMS (Evaluation Decision Making System) ◦ ◦ The EDMS is a centralized public access portal designed to provide information related to evaluation of a library's services and resources for management and advocacy purposes. Information provided includes: ◦ Types of evaluation methods typically used to assess the use of services and resources; ◦ Data each type of evaluation can provide; ◦ How to plan for data collection efforts and tips on how to analyze the data; and ◦ Strategies on how to apply the results of evaluation efforts for management and advocacy purposes.