Seeking Synchronicity: Evaluating Virtual Reference Transcripts Presented by Lynn Silipigni Connaway OCLC Members Council February 14, 2006

Seeking Synchronicity : Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives $1,103,572 project funded by: –Institute of Museum and Library Services $684,996 grant –Rutgers, The State University of New Jersey and OCLC Online Computer Library Center $405,076 in kind contributions

Seeking Synchronicity : Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives Project duration 10/1/2005-9/30/2007 Four phases: I.Focus group interviews* II.Analysis of 1,000 QuestionPoint transcripts III.600 online surveys* IV.300 telephone interviews* *Interviews & surveys with VRS users, non-users, & librarians

Phase II: 24/7 Transcript Analysis Generated random sample – July 7, 2004 through June 27, 2005 – 263,673 sessions – 25 transcripts/month = 300 total 256 usable transcripts – Excluding system tests and technical problems

5 Analyses Geographical Distribution Type of Library Type of Questions Katz/Kaske Classification Subject of Questions Interpersonal Communication Radford Classification Manual and NVivo coding

Geographical Distribution of Question Origination

Type of Library

Type of Questions Combined 1 st and 2 nd Questions n=273 questions

Subjects of Questions Combined 1 st and 2 nd Questions n=273 questions

Classification Methodology Qualitative Analysis Development/refinement of category scheme Careful reading/analysis Identification of patterns Time intensive, but reveals complexity

Research Questions Interpersonal Communication Analysis What relational dimensions are present? Are there differences in relational dimensions/patterns of chat users & librarians? If so, what are they? How do users & librarians compensate for lack of nonverbal cues? What is the relationship between content & relational dimensions in determining quality?

2 Major Themes Relational Facilitators Interpersonal aspects of the chat conversation that have a positive impact on the librarian- client interaction and that enhance communication. Relational Barriers Interpersonal aspects of the chat conversation that have a negative impact on the librarian- client interaction and that impede communication. Results Interpersonal Communication Analysis

Manual Coding Results 200 Transcripts – 177 Usable Transcripts

Librarian Relational Factors (1- 200) Librarian Relational Facilitators: Manual Analysis n=177 transcripts

Client Relational Facilitators: Manual Analysis n=177 transcripts

Comparison Relational Facilitators: Manual Analysis n=177 transcripts

Librarian Relational Barriers: Manual Analysis n=177 transcripts

Client Relational Barriers: Manual Analysis n=177 transcripts

Comparison of Relational Barriers: Manual Analysis n=177 transcripts

NVivo Coding 100 Transcripts – 79 Usable Transcripts

Librarian Relational Facilitators: NVivo Analysis n=79 transcripts

Client Relational Facilitators: NVivo Analysis n=79 transcripts

Comparison Relational Facilitators: NVivo Analysis n=79 transcripts

Librarian Relational Barriers: NVivo Analysis n=79 transcripts

Client Relational Barriers: NVivo Analysis n=79 transcripts

Comparison Relational Barriers: NVivo Analysis n=79 transcripts

NVivo Search Totals WordTotalPercent sorry thank you thanks thank u33.80 n=79 transcripts Includes words in scripts

NVivo Search Totals WordTotalPercent sorry thank you thanks thank u33.80 thanx00.00 TY00.00 ty00.00 n=79 transcripts Includes words in scripts

NVivo Search Totals PersonWordTotalPercent Librariansorry Clientsorry78.86 Librarianthank you Librarianthanks Clientthank you Clientthanks Clientthank u33.80 n=79 transcripts Includes words in scripts

NVivo 2.0: Advantages Advantages –Move subnodes between nodes at any time Ability to create a hierarchical classification scheme –Proximity and Boolean searching –Group documents and nodes into sets for use in searching and analysis –Programmatically calculate descriptive statistics after coding

NVivo 2.0: Disadvantages Disadvantages –User interface does not match standard Microsoft application Cannot double click on a document or node to open Cannot scroll through a document using the mouse scroll wheel –Cannot import nodes –Cannot set certain tools to always appear Must manually display the Coder bar and coding strips each time for every document/transcript NVivo 7 to be released in Feb –May alleviate disadvantages

Transcript Reading Positive VRS experience –Duration = 1 hour 11 minutes –Academic User –Question – Boston drug company - diabetes –Relational Work –Enthusiastic user –Helpful librarian Less than positive VRS experience –Duration = 39 minutes –Middle school or high school student –Question – physics – car acceleration –Poor reference work –Extreme negative closure

End Notes This is one of the outcomes from the project Seeking Synchronicity: Evaluating Virtual Reference Services from User, Non-User, and Librarian Perspectives, Marie L. Radford and Lynn Silipigni Connaway, Co-Principal Investigators. Funded by IMLS, Rutgers University and OCLC, Online Computer Library Center. Project web site: onicity/ onicity/

Questions Marie L. Radford, Ph.D. Lynn Silipigni Connaway, Ph.D.