1. The AAHSL/ARL Partnership in Exploring Outcomes Assessment through LibQUAL+

2. Surfing the Tsunami of Service Quality2 Tamera Lee, Director, Greenblatt Library, Medical College of Georgia, Chair, AAHSL Task Force on Service Quality Assessment James Shedlock, Director, Galter Health Sciences Library, Northwestern University, Editor of AAHSL Annual Statistics, Task Force Liaison Rick Forsman, Director, Denison Memorial Library, University of Colorado Health Sciences Center, AAHSL President 2002/Board Liaison

3. Surfing the Tsunami of Service Quality3 What is LibQUAL + A new measurement initiative of the Association of Research Libraries and researchers at TAMU (Colleen Cook, Fred Heath and Bruce Thompson) Define quality to complement the traditional quantitative library measures (number of volumes, amount of space, size of staff, etc.)

4. Surfing the Tsunami of Service Quality4 What is LibQUAL+? National, broad-scale project to define and measure service quality across institutions and identify best practices Web-based survey based on research grounded in gap analysis from the service quality business literature Parasuraman, Zeithaml and Berry

5. Surfing the Tsunami of Service Quality5 What is LibQUAL+? Specifically the survey indicates how user expectations compare with user perceptions of service quality Measure gaps between what service dimensions users most desire, what they expect and perceive they are getting

6. Surfing the Tsunami of Service Quality6 Purpose/Task Force Outline the qualitative efforts of the AAHSL Task Force on Service Quality Assessment: Jane Blumenthal- Georgetown University Diana Cunningham - NY Medical College Rick Forsman - University of Colorado Tamera Lee - Medical College of Georgia Logan Ludwig - Loyola University-Chicago James Shedlock -Northwestern University Julie Sollenberger - University of Rochester Susan Starr - University of California, San Diego Laurie Thompson -State University of New York, Upstate

7. Surfing the Tsunami of Service Quality7 The Association of Academic Health Sciences Libraries (AAHSL) One of two consortia’s in the 2002 pilot of LibQUAL+™; the other was the Ohio Library and Information Network (OhioLINK), a consortium of Ohio’s college and university libraries and the State Library of Ohio. Focus: the AAHSL/ARL partnership, the AAHSL aggregate experience, results and collaborative outcomes.

8. Surfing the Tsunami of Service Quality8 Background With a long history of compiling and analyzing comparative statistics to serve as useful benchmarks for member libraries, AAHSL recognizes the increasing importance of outcomes measures to be used as performance indicators.

9. Surfing the Tsunami of Service Quality9 Background AAHSL partnered with ARL for the participation of a cross section of 36 AAHSL libraries in the spring 2002 pilot. The AAHSL project was funded generously by the National Library of Medicine, with shared costs by AAHSL and participating libraries to support deliverables.

10. Surfing the Tsunami of Service Quality10 Background The Task Force on Quality Assessment was created in July 2001 to lead AAHSL’s involvement in a service quality assessment process and evaluate LibQUAL+™ as a qualitative measurement tool for academic health sciences libraries.

11. Surfing the Tsunami of Service Quality11 Task Force Goals Testing the usefulness of the instrument for benchmarking Testing the feasibility of customizing the instrument for the health sciences Determining the value of cross data analysis with AAHSL statistics Gaining experience with an aggregate outcomes measurement process

12. Surfing the Tsunami of Service Quality12 The Process Coordinated the cohort registration and established a listserv to facilitate efficient communication Drafted a procedural manual to serve as a guide and created a survey administration tip sheet for participants

13. Surfing the Tsunami of Service Quality13 Task Force Challenges Clarified the importance of a clean set of collective data for AAHSL and defined demographics for respondents

14. Surfing the Tsunami of Service Quality14 Task Force Challenges Developed 5 unique AAHSL questions in collaboration with participants and in consultation with the National Library of Medicine. The main objective was to add service performance issues for the health sciences, especially priorities not included or covered sufficiently by the LibQUAL+™ survey.

15. Surfing the Tsunami of Service Quality15 Unique AAHSL Queries Providing health information when and where I need it Employees teaching me how to access or manage information Environment that facilitates group study and problem solving Access to information resources that support patient care Having comprehensive electronic resources

16. Surfing the Tsunami of Service Quality16 Library Service Quality Survey LibQual+ When it comes to… My Minimum Service Level Is low high My Desired Service Level Is low high Perceived Service Performance Is low high N/A 1) Willingness to help users 1 2 3 4 5 6 7 8 9 N/A 2) Space that facilitates quiet study 1 2 3 4 5 6 7 8 9 N/A 3) Complete runs of journal titles 1 2 3 4 5 6 7 8 9 N/A

17. Surfing the Tsunami of Service Quality17 Task Force Challenges Guided discussion of human subjects clearance and disparities of IRBs regarding issues of anonymity, age, etc.

18. Surfing the Tsunami of Service Quality18 The Respondents The TF recommended that the AAHSL participants survey their entire user populations. Only those who completed the survey in its entirety and with measurable consistency were included. Over half (50.97%) of the people who entered the survey completed it. With each of the 4 pages viewed the percentage dropped 8-20%, most significantly at the 2 nd (demographics) and 3 rd pages.

19. Surfing the Tsunami of Service Quality19 Results Chart—percentage of AAHSL respondents by user group (N=13,976)

20. Surfing the Tsunami of Service Quality20 Results Chart—percentage of respondents for disciplines reflecting 5% or more (N=13,976)

21. Surfing the Tsunami of Service Quality21 Results Chart—average scores for AAHSL aggregate rating general satisfaction from 1-9 (with 1 being “strongly disagree” and 9 representing “strongly agree”)

22. Surfing the Tsunami of Service Quality22 Frequency of Library Use—AAHSL Aggregate Most respondents reported using the Library weekly, both on the premises (35.93%) and electronically (34.84%). Daily electronic use was reported by 31.83% and monthly use on premises was reported by 31.42%.

23. Surfing the Tsunami of Service Quality23 Service Dimension Gaps that fell outside “zone of tolerance” AAHSL aggregate—electronic resources accessible from home or office AAHSL students—convenient business hours AAHSL faculty—complete runs of journal titles, electronic resources accessible from home or office, comprehensive electronic resources

24. Surfing the Tsunami of Service Quality24 Dimensions Most Desired by AAHSL Aggregate Comprehensive electronic resources (8.44) Electronic resources accessible from home or office (8.35)

25. Surfing the Tsunami of Service Quality25 Dimensions Most Desired by AAHSL Aggregate Students : ( undergrad and grad): having comprehensive electronic resources (8.33- 8.50), followed closely by convenient business hours (8.32-8.43) Faculty : having comprehensive electronic resources (8.49), followed closely by making electronic resources accessible from home or office (8.46)

26. Surfing the Tsunami of Service Quality26 Service Affect Dimensions Most Desired by Staff Employees who are courteous consistently (8.55), Willingness to help users (8.52) Employees who have the knowledge to answer users’ questions (8.46) Readiness to respond to users’ questions (8.46)

27. Surfing the Tsunami of Service Quality27 The Results ARL reliability results indicate higher AAHSL user expectations and levels of satisfaction compared to the ARL environment. Responses from the general academic campus reflect a higher response rate from undergraduate students, whose expectations and needs differ from the user profile of AAHSL institutions, consisting primarily of graduate level and above.

28. Surfing the Tsunami of Service Quality28 The Results Useful view of the data is through T-scores and norm tables. T-scores enables examination of individual score norms for faculty and students and institutional scores in relation to scores of peer institutions.

29. Surfing the Tsunami of Service Quality29 The Results The overall norm table for institutions of all types shows that one health sciences library clearly stands out as exceeding expectations-- even desires. Mercer University School of Medicine Library and LRC has the only positive average superiority score, indicating overall perceptions are higher than desired.

30. Surfing the Tsunami of Service Quality30 Comments Frequent strong themes shared included 1) need for more journals, especially online 2) frustration with remote access problems 3) positive comments about staff

31. Surfing the Tsunami of Service Quality31 AAHSL Queries “Comprehensive Electronic Resources ” Reflected the highest desired measure by all user groups Represented a significant gap in minimal expectations for faculty

32. Surfing the Tsunami of Service Quality32 AAHSL Queries “Comprehensive Electronic Resources” The ARL research team advised the task force to reconsider this question as an unreasonable service demand. Seemingly the sense is that comprehensive print resources may be more achievable. For the 2003 LibQUAL+™ cohort iteration, this question was revised to provide more focus, “Electronic resources matching my information needs.”

33. Surfing the Tsunami of Service Quality33 AAHSL Queries Summative data for other four AAHSL queries indicates needs for slight improvements for all user groups.

34. Surfing the Tsunami of Service Quality34 Participating Libraries - Outcomes Data used to complement other assessment measures and create additional ones, such as focus groups. A correlative test to consider the integration of qualitative and quantitative measures conducted by Lee, Dennison and Joubert showed no correlation between service affect gap score and annual expenditures.

35. Surfing the Tsunami of Service Quality35 Participating Libraries- Outcomes Many shared presentations with respective senior administrations, academic senates, student government associations, physical facilities planners and library staff. Some made immediate changes and others want to repeat the survey in 2003 for a better view. A few incorporated LibQUAL+™ data into their annual performance metrics and other review processes.

36. Surfing the Tsunami of Service Quality36 Task Force Recommendations for AAHSL: Participate in the final phase of pilot (2003 survey administration) for a cross section of 23 new and repeat participating libraries to optimize our assessment and maximize our deliverables

37. Surfing the Tsunami of Service Quality37 Task Force Recommendations for AAHSL Consider if and how a qualitative data component (such as LibQUAL+ survey results) may be used in conjunction with the annual quantitative stats for member libraries Continue to support LibQUAL+ assessment and development, other qualitative measurement activities and future initiatives in outcomes assessment

38. Surfing the Tsunami of Service Quality38 Benefits-ARL Assisted in furthering the applicability of LibQUAL+™ as a qualitative assessment tool across different types of institutions. The special cohort assisted ARL with promotion, training and increased participation.

39. Surfing the Tsunami of Service Quality39 Benefits-AAHSL Partnership created a heightened awareness of the viability of outcomes assessment and a cost efficient protocol for listening to users. The additional five questions allowed AAHSL to include a construct for issues perceived as unique or more significant to health sciences libraries. Prompted further interest in testing the instrument's feasibility for a new cohort of hospital libraries. Advanced a culture of assessment within AAHSL and for its members.