1. Academic Libraries and Virtual Reference User Preferences & Usability of Reference Services July 12, 2011 Rebecca Croxton, MLIS Anthony S. Chow, Ph.D.

2. Purpose of Study Better understand information seeking preferences of library users in an academic setting Develop an understanding of user perceptions of usability of different virtual reference mediums offered by an academic library

3. Research Questions 1.Which virtual reference mediums do library reference users prefer? (Email, Telephone, Online Chat, Video Conferencing, Text Messaging) 2.Are there differences in user preferences for virtual reference mediums based on type of question? 3.Are there differences in user preferences for virtual reference mediums based on type of user and type of question? 4.Which virtual reference mediums do library users find most usable?

4. Findings in the Literature Findings vary across studies Washington State University (2 campuses) (N=197) South Atlantic (2 Universities) (N=276) 1 st Choice When Seeking Library Assistance 1 st Choice When Seeking Reference Help with Research Library Web Site (42.4%)Face-to-Face (66.4%) Reference Librarian (14.5%)Email (20.2%) Friend (10.9%)Telephone (9.2%) Online Chat (10.3%)Online Chat (4.2%) Telephone (10.3%) Email (7.2%) Other (3.6%) Joel Cummings, Lara Cummings, and Linda Frederiksen, "User Preferences in Reference Services: Virtual Reference and Academic Libraries," Portal 7, no. 1 (January 2007). Corey M. Johnson, "Online Chat Reference," Reference & User Services Quarterly 43, no. 3 (Spring 2004): 241.

5. Study Methodology Campus-wide Survey (936 participants) Usability Study (31 university student participants) Focus Group of Reference Library Staff (13 participants) Interview with Head of Reference at Academic Library

6. Participants – Campus Wide Survey Campus-Wide Survey - 936 respondents Sent to a randomly selected sample of university students (N=2,552 / 14.2% response rate) All permanent faculty and staff (N=3,840 / 14.3% response rate)

7. Instrumentation – Campus Wide Survey 11-item online survey Questions: –Prior experience with and knowledge of reference services –Rank order virtual reference seeking preferences for answering a procedural based, research question and a quick, factual question –Choose one reference service for procedural, research question and quick, factual question – this time including face- to-face reference as option

8. Key Findings – Campus-Wide Survey Fifty-four percent of respondents had prior experience with at least one reference medium.

9. Key Findings - Campus-Wide Survey Research Questions Email is most popular choice for research questions.

10. Key Findings – Campus-Wide Survey Research Questions If face-to-face is an option, survey respondents prefer face-to-face for research questions. Faculty prefer email over face-to-face reference. After face-to-face, undergraduates nearly equally prefer email and online chat.

11. Key Findings – Campus-Wide Survey Quick, Factual Questions Telephone is most popular for quick, factual questions.

12. Key Findings – Campus-Wide Survey Quick, Factual Questions With face-to-face as an option, telephone is most preferred for factual questions. Online chat and telephone were nearly equally preferred for undergraduates. Faculty preferred email over all other choices.

13. Participants – Usability Study Usability Study - 31 participants 12 Randomly Selected Undergraduate Students 19 Randomly Selected Graduate Students Of these participants: 15 Students Tested Reference Mediums at University A 16 Students Tested Reference Mediums at University B Universities Utilized for Study: Two mid-sized, doctoral granting state universities in the Southeast

14. Instrumentation – Usability Study Participants given written instructions via email to contact a designated university library and ask a pre-assigned procedural, research based question and a quick factual question via: –Email –Telephone –Online Chat –Skype Video* –Text Messaging * University B did not offer Skype Video Reference

15. Instrumentation – Usability Study (Cont.) Participants completed online surveys after testing each medium and at completion of study –Rank order virtual reference seeking preferences for answering a procedural based, research question and a quick, factual question –Choose one reference service for procedural, research question and quick, factual question – this time including face- to-face reference as option –Rate usability of virtual reference mediums using measures defined by ISO 9241-11:1998 of Effectiveness, Efficiency, and SatisfactionISO 9241-11:1998

16. Key Findings – Usability Study Research Questions Online chat is most preferred for research questions.

17. Key Findings – Usability Study Research Questions With face-to-face as an option, online chat is still most preferred for research questions

18. Key Findings – Usability Study Quick, Factual Questions Online chat is most preferred for quick, factual questions.

19. Key Findings – Usability Study Quick, Factual Questions With face-to-face as an option, online chat is still most preferred for quick, factual questions.

20. Key Findings – Research Questions Campus-Wide Survey –Email is most preferred virtual reference medium –When face-to-face reference is an option, users prefer face-to-face over all other choices Exception – Faculty preferred email over F2F Usability Study (students only) –Online chat is preferred reference medium over all other choices (including face-to-face)

21. Key Findings – Quick, Factual Questions Campus-Wide Survey –Telephone is most preferred virtual reference medium –When face-to-face reference is an option, users prefer telephone over all other choices –Exceptions: –Faculty prefer email over face-to-face reference –Students equally prefer telephone and online chat Usability Study (students only) –Online chat is preferred reference medium over all other choices (including face-to-face)

22. Summary of Preferences Research Question 1 st Choice Quick, Factual Question 1 st Choice Virtual Reference Overall (w/ Face-to- Face) Virtual Reference Overall (w/ Face-to- Face) All Users Combined EmailFace-to-FaceTelephone UndergraduatesOnline ChatFace-to-Face* Telephone* (followed closely by Online Chat) Online Chat Graduate Students Email*Face-to-Face* Telephone* (followed closely by Online Chat) FacultyEmail Email (followed closely by F2F) Telephone (followed closely by Email) Email StaffEmailFace-to-FaceTelephone *In usability study, both graduate and undergraduate students rated Online Chat as their first choice, regardless of the availability of Face-to-Face Reference.

23. Usability - Defined Usability is a quality attribute relating to how easy something is to use. … it refers to how quickly people can learn to use something, how efficient they are while using it, how memorable it is, how error- prone it is, and how much users like using it. If people cant or wont use a feature, it might as well not exist. J. Nielsen & H. Loranger, Prioritizing Web Usability. (Berkeley, CA: New Riders, 2006).

24. Usability - Operationalized Per ISO 9241-11:1998, usability is measured using three factors:ISO 9241-11:1998, Efficiency Effectiveness Satisfaction

25. Usability - Effectiveness Effectiveness (Jordan, 2001) –Task Completion: Were you able to successfully use [reference medium] to ask your questions and have them answered? –Quality of Output: How satisfied were you with the overall quality of the [reference medium] responses and the extent to which they answered your questions? P.W. Jordan, An Introduction to Usability. (Philadelphia, PA: Taylor & Francis, 2001). Effectiveness – Usability Ratings (1=lowest satisfaction / 10=highest satisfaction) Task Completion Quality of Output Overall Effectiveness Email (n=31)8.296.037.16 Telephone (n=30)8.778.538.65 Online Chat (n=30)9.509.709.60 Skype Video (n=15)*6.336.076.20 Text Messaging (n=29)6.216.006.10 *Skype reference was only available at University A

26. Usability - Efficiency Efficiency (Jordan, 2001) –Deviations from Critical Path: How satisfied are you that you were able to use [reference medium] without any problems? –Error Rate: How satisfied were you that you accomplished your goal with very few or no errors? –Time-on-Task: How satisfied are you that you accomplished your goal with minimal time wasted? –Mental Effort: How satisfied are you that you accomplished your goal with minimal mental effort? Efficiency – Usability Ratings (1=lowest satisfaction / 10=highest satisfaction) Deviations from Critical Path Error Rate Time-on- Task Mental Effort Overall Efficiency Email (n=31)8.658.777.268.458.28 Telephone (n=30)9.07 8.038.538.68 Online Chat (n=30)9.49.479.339.279.37 Skype Video (n=15)*5.075.475.275.205.25 Text Msg (n=29)6.456.526.006.486.36 *Skype reference was only available at University A

27. Usability - Satisfaction Satisfaction (Jordan, 2001) –Overall, how satisfied were you with the general usability of using [reference medium] to ask reference questions?

28. Usability – Overall Results Overall, research subjects found online chat to the be most usable virtual reference medium. Overall Usability Ratings (1=lowest satisfaction / 10=highest satisfaction) Overall Effectiveness Overall Efficiency Overall Satisfaction Overall Usability (mean) Email (n=31)7.168.288.037.82 Telephone (n=30)8.658.688.708.68 Online Chat (n=30)9.609.379.609.52 Skype Video (n=15)*6.205.255.675.71 Text Messaging (n=29)6.106.366.386.28 *Skype reference was only available at University A

29. Instrumentation – Focus Group & Interview Focus Group Reference Library Staff (n=13) at University A 7 Questions re: thoughts, preferences, and future trends surrounding virtual reference Sample Question: –Based on your experience, which of the reference mediums (email, telephone, online chat, Skype video, or text messaging) is most popular with library patrons? Interview with Reference Dept Manager 18 open-ended questions re: thoughts & opinions re: virtual reference services

30. Key Findings – Focus Group Online chat is without a doubt the most popular reference medium choice for library patrons both for quick factual questions and research questions. If individuals already have established relationships with particular librarians (such as a subject librarian or embedded in a course), email seems to be more popular. Usage of online chat and email fluctuates depending on the time of day and time frame in which a library patron needs a response. The needs of the users drive the transaction.

31. Key Findings – Interview with Reference Manager In the past 3 ½ years… Telephone inquiries have gone down dramatically. Online Chat inquiries have increased substantially (330% increase since 2005). Email inquiries at reference desk have decreased, but emails with departmental liaisons have increased.

32. Answering the Research Questions 1.Which virtual reference mediums do library reference users prefer? –Results suggest library user preferences are dependent upon type of question and type of user. Overall for virtual reference, findings suggest users prefer Email for research questions and Telephone for quick, factual questions. 2.Are there differences in user preferences for virtual reference mediums based on type of question? –Yes! Preferences differ significantly depending upon type of question. 3.Are there differences in user preferences for virtual reference mediums based on type of user and type of question? –Yes! Preferences differ significantly depending upon type of question and type of user. 4.Which reference mediums do library users find most usable? –Study findings suggest users find Online Chat to be the most usable virtual reference medium.

33. Explaining the Results Pooles Principle of Least Effort. –Information seekers will attempt to minimize the overall work associated with seeking information and will seek the most convenient source to meet their information needs at any given time. A persons role may determine the depth of an information need, thereby influencing their choice of reference medium. Familiarity, convenience, and ease-of-use appear to be primary factors behind information seeking preferences. Herbert Poole, Theories of the Middle Range (Norwood, NJ: Ablex, 1985).

34. Using Factor Analyses to Help Interpret the Results Factor Analysis of Campus Wide Survey participants qualitative comments suggest: –Email – primary factors behind preferences focused on: Convenience Familiarity Ease of Use Written Record that was precise and in-depth –Online Chat, Telephone, and Text Messaging – primary factors behind preferences focused on Speed and Quickness of Response Convenience Familiarity Ease of Use

35. What Next? Replicate study with other populations to see if findings remain consistent. Watch the trends for video conferencing and text messaging reference, particularly among younger populations. Millennial generation will start becoming faculty and staff at universities.

36. References Joel Cummings, Lara Cummings, and Linda Frederiksen, "User Preferences in Reference Services: Virtual Reference and Academic Libraries," Portal 7, no. 1 (January 2007). Corey M. Johnson, "Online Chat Reference," Reference & User Services Quarterly 43, no. 3 (Spring 2004): 241. J. Nielsen & H. Loranger, Prioritizing Web Usability. (Berkeley, CA: New Riders, 2006). P.W. Jordan, An Introduction to Usability. (Philadelphia, PA: Taylor & Francis, 2001). Herbert Poole, Theories of the Middle Range (Norwood, NJ: Ablex, 1985).