How to Evaluate a Library Service: Methods and Examples Sarah Aerni Special Projects Librarian University of Pittsburgh Aernis@yahoo.com 1 April 2005

Outline of this lecture An overview of the University of Pittsburgh Library System How we evaluated our faculty Document Delivery Service for Faculty

Hillman Library - The Main Library of the University of Pittsburgh

Hillman Library

Hillman Library - The first floor reading area and looking across to the East Asian Library

The East Asian Library of the University of Pittsburgh Supports Asian Studies Program for Chinese, Japanese and Korean Studies 280,000 volumes of Chinese materials 58,000 volumes of Japanese materials 3,200 volumes of Korean materials 1,900 Periodical titles (over 45,000 volumes bound); Contains titles published in Mainland China, Hong Kong, Taiwan, Japan and Korea For more Information: http://www.library.pitt.edu/libraries/eal/index.htm

The Libraries to Go Service This is a document delivery service for faculty members only. The faculty search for and request the library materials they need. We obtain the material and bring it to them in their office (for books) or send it electronically (for journal articles). We promise to deliver the material within 24-48 hours after they request it. Requests must be made electronically through a web form.

Delivery Boundaries

Evaluation Goals We began offering this service in April 2003. In April of 2004, the administration wanted to evaluate the following questions: Who is using the service? Who is not using the service? Are the users satisfied with the service? Have they had any problems with it?

Soliciting Participants for the Evaluation We decided to ask faculty members who have used the service about their experiences with it. The format for the survey was a web form. An email was sent out to 162 people asking for them to fill out the web survey. Responses were submitted to me and I compiled them. This method did not allow us to question non-users of the service.

The Survey (1)   How satisfied were you with the Libraries to Go Service? Please answer on a scale of 1-5 (2)   Why or why not? (3)   Have you encountered any problems using the service? How long did it take you to fill out the form and understand the proper procedures for using the Libraries to Go service? (4)   How did you hear about the Libraries to Go Service? (5)   Did you receive the requested materials within 24-48 hours? Please feel free to add any additional comments about the Libraries to Go service.

Notes about the Survey The survey only has 6 main questions. We kept it short so that it was easy to fill out. Only 2 questions are text, which helps to reduce the time necessary to complete it. This is a “quick and dirty” method of obtaining information about usage.

Response Rate We sent the request to participate out to 162 people. 8 email addresses were invalid. We received 51 responses. This is a 31.5% response rate.

Results –1 The first question addressed the issue of personal satisfaction with the service. Satisfaction was measured on a scale of 1-5. The average response was 4.78/5.00 satisfaction level. Why is this so high?

Results - 2 We asked if the material had arrived in their office within 24-48 hours after making the request. 86% said that it had come on time. 14% said that it did not come on time. The comments which people sent helped us determine why the materials did not always arrive on time.

Positive comments from users “This service saves me time and reduces the cost to me of getting books from the library system. If I had to go and fetch the books from all the different libraries myself, I would probably not use as many of them and my research would suffer as a result. “

Positive comments from users “Service was quick, useful and made life easier for me.” “It saves me a great deal of time to have books delivered to my office. The service has been dependable.”

Changes we made based on the Evaluation We changed the boundary map to include some new parts of campus. We made the due date for the information to be returned more clear. We decided to promote the service more to new faculty and others who may not have heard about it. We decided to lend video and audio materials through this service too.

Thank you for coming! Please feel free to ask anything further about this survey or about the University of Pittsburgh.