1. Seeking Synchronicity: Evaluating Virtual Reference Transcripts Presented by Lynn Silipigni Connaway OCLC Members Council February 14, 2006

2. Seeking Synchronicity : Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives $1,103,572 project funded by: –Institute of Museum and Library Services $684,996 grant –Rutgers, The State University of New Jersey and OCLC Online Computer Library Center $405,076 in kind contributions

3. Seeking Synchronicity : Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives Project duration 10/1/2005-9/30/2007 Four phases: I.Focus group interviews* II.Analysis of 1,000 QuestionPoint transcripts III.600 online surveys* IV.300 telephone interviews* *Interviews & surveys with VRS users, non-users, & librarians

4. Phase II: 24/7 Transcript Analysis Generated random sample – July 7, 2004 through June 27, 2005 – 263,673 sessions – 25 transcripts/month = 300 total 256 usable transcripts – Excluding system tests and technical problems

5. 5 Analyses Geographical Distribution Type of Library Type of Questions Katz/Kaske Classification Subject of Questions Interpersonal Communication Radford Classification Manual and NVivo coding

6. Geographical Distribution of Question Origination

7. Type of Library

8. Type of Questions Combined 1 st and 2 nd Questions n=273 questions

9. Subjects of Questions Combined 1 st and 2 nd Questions n=273 questions

10. Classification Methodology Qualitative Analysis Development/refinement of category scheme Careful reading/analysis Identification of patterns Time intensive, but reveals complexity

11. Research Questions Interpersonal Communication Analysis What relational dimensions are present? Are there differences in relational dimensions/patterns of chat users & librarians? If so, what are they? How do users & librarians compensate for lack of nonverbal cues? What is the relationship between content & relational dimensions in determining quality?

12. 2 Major Themes Relational Facilitators Interpersonal aspects of the chat conversation that have a positive impact on the librarian- client interaction and that enhance communication. Relational Barriers Interpersonal aspects of the chat conversation that have a negative impact on the librarian- client interaction and that impede communication. Results Interpersonal Communication Analysis

13. Manual Coding Results 200 Transcripts – 177 Usable Transcripts

14. Librarian Relational Factors (1- 200) Librarian Relational Facilitators: Manual Analysis n=177 transcripts

15. Client Relational Facilitators: Manual Analysis n=177 transcripts

16. Comparison Relational Facilitators: Manual Analysis n=177 transcripts

17. Librarian Relational Barriers: Manual Analysis n=177 transcripts

18. Client Relational Barriers: Manual Analysis n=177 transcripts

19. Comparison of Relational Barriers: Manual Analysis n=177 transcripts

20. NVivo Coding 100 Transcripts – 79 Usable Transcripts

21. Librarian Relational Facilitators: NVivo Analysis n=79 transcripts

22. Client Relational Facilitators: NVivo Analysis n=79 transcripts

23. Comparison Relational Facilitators: NVivo Analysis n=79 transcripts

24. Librarian Relational Barriers: NVivo Analysis n=79 transcripts

25. Client Relational Barriers: NVivo Analysis n=79 transcripts

26. Comparison Relational Barriers: NVivo Analysis n=79 transcripts

27. NVivo Search Totals WordTotalPercent sorry2025.32 thank you5164.56 thanks3443.04 thank u33.80 n=79 transcripts Includes words in scripts

28. NVivo Search Totals WordTotalPercent sorry2025.32 thank you5164.56 thanks3443.04 thank u33.80 thanx00.00 TY00.00 ty00.00 n=79 transcripts Includes words in scripts

29. NVivo Search Totals PersonWordTotalPercent Librariansorry1417.72 Clientsorry78.86 Librarianthank you3645.57 Librarianthanks1822.78 Clientthank you2835.44 Clientthanks2430.38 Clientthank u33.80 n=79 transcripts Includes words in scripts

30. NVivo 2.0: Advantages Advantages –Move subnodes between nodes at any time Ability to create a hierarchical classification scheme –Proximity and Boolean searching –Group documents and nodes into sets for use in searching and analysis –Programmatically calculate descriptive statistics after coding

31. NVivo 2.0: Disadvantages Disadvantages –User interface does not match standard Microsoft application Cannot double click on a document or node to open Cannot scroll through a document using the mouse scroll wheel –Cannot import nodes –Cannot set certain tools to always appear Must manually display the Coder bar and coding strips each time for every document/transcript NVivo 7 to be released in Feb. 2006 –May alleviate disadvantages

32. Transcript Reading Positive VRS experience –Duration = 1 hour 11 minutes –Academic User –Question – Boston drug company - diabetes –Relational Work –Enthusiastic user –Helpful librarian Less than positive VRS experience –Duration = 39 minutes –Middle school or high school student –Question – physics – car acceleration –Poor reference work –Extreme negative closure

33. End Notes This is one of the outcomes from the project Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives, Marie L. Radford and Lynn Silipigni Connaway, Co-Principal Investigators. Funded by IMLS, Rutgers University and OCLC, Online Computer Library Center. Project web site: http://www.oclc.org/research/projects/synchr onicity/ http://www.oclc.org/research/projects/synchr onicity/

34. Questions Marie L. Radford, Ph.D. Email: mradford@scils.rutgers.edumradford@scils.rutgers.edu www.scils.rutgers.edu/www.scils.rutgers.edu/~mradford Lynn Silipigni Connaway, Ph.D. Email: connawal@oclc.org www.oclc.org/research/staff/connaway.htm