1. Qualitative Research Methods for Assessing Services and Programs in Academic Libraries Dr. Robert V. Labaree Applied Social Sciences Librarian University of Southern California labaree@usc.edu CARL 12 th Biennial Conference Hyatt Regency, Irvine, California Hyatt Regency, Irvine, California April 2-5, 2008

2. Introduction Topics to be covered: Methodological features of qualitative studies Methodological features of qualitative studies When to apply qualitative methods in assessing library services and programs When to apply qualitative methods in assessing library services and programs Some techniques for gathering data in library settings Some techniques for gathering data in library settings Benefits and pitfalls associated with qualitative research Benefits and pitfalls associated with qualitative research Recommended resources Recommended resources Conclusion and contact information Conclusion and contact information Slide 2 of 20

3. General Design Characteristics of Qualitative Research  Naturalistic Investigate social processes as they unfold naturally rather than reasoning from either the conditions under which they occur or the outcomes that correlate with them.  Emergent Willingness to adapt methods of analysis as knowledge and understanding deepens  Purposeful Cases are selected because they are “information rich” Source: Patton, Michael Q. Qualitative Research and Evaluation Methods. 3 rd ed. (Thousand Oaks, CA: Sage, 2002), pp. 40- 41. [and next two slides] Slide 3 of 20

4. Data Collection Characteristics of Qualitative Research  “Thick description” The goal is deep understanding about phenomena and experiences  Personal experience and engagement The researcher is the primary instrument of data collection and analysis  Empathic stance Sensitivity, respect, awareness, responsiveness, and openness  Attention to process Situational and contextual changes occur throughout the life of the study Slide 4 of 20

5. Analytical Characteristics of Qualitative Research  Unique case orientation Every case is special and unique  Inductive and creative Research process guided by analytical principles rather than rules  Holistic perspective Phenomena under study are complex and cannot be reduced to a few variables  Context sensitivity Avoid broad generalizations; investigation does not rest in isolation from the larger social/cultural/economic/political environment  Reflexivity Critical self-awareness of one’s own presence in the research process Slide 5 of 20

6. Selective Applications in Library Settings  To assess individualized outcomes  To explore internal dynamics of programs and services  To evaluate critical cases of program or service utilization  To acquire details about program implementation  To obtain information about the nuances of program or service quality  To personalize the evaluation process and/or outcomes assessment  To discover effects of specific programs or services  To add depth, detail, and meaning to statistical data  To provide privileged access to processes, causes, and effects. Source: Patton, Michael Q. How to Use Qualitative Methods in Evaluation. 1 st ed. (Thousand Oaks, CA: Sage, 1987), pp. 41-42. Slide 6 of 20

7. Some Techniques for Collecting Assessment Data In-Depth Interviews – speaking with and listening to our users Observation – witness what is going on in a particular setting and explore the experiences of others Qualitative Case Studies – holistic understanding of social phenomena in library settings Slide 7 of 20

8. In-Depth Interviews Purpose In-depth interviewing seeks to achieve the same level of knowledge and understanding possessed by the respondent and to understand personal experiences and perceptions within a contextualized, social framework. Johnson, John M. “In-Depth Interviewing.” In Handbook of Interview Research: Context and Method. Jaber F. Gubrium and James A. Holstein, eds. (Thousand Oaks, CA: Sage), pp. 103-119. 8 of 20

9. In-Depth Interviews Design Considerations General Typologies Structured, semi-structured, conversational Seven Stages of an Interview Investigation (Kvale, 1996) 1. Thematizing—constructing the study’s purpose 2. Designing—protocol of how interview is to be undertaken 3. Interviewing—reflexive and ethical 4. Transcribing—preparing data for analysis 5. Analyzing—coding the data and discovering themes 6. Verifying—ascertain validity and reliability 7. Reporting—communicate findings to stakeholders 9 of 20

10. In-Depth Interviews Outcomes for Assessment What is meaningful or important is in the respondent’s own voice What is meaningful or important is in the respondent’s own voice Researcher’s knowledge, expertise, and interpersonal skills can be used to explore interesting or unexpected concepts or issues raised by respondents Researcher’s knowledge, expertise, and interpersonal skills can be used to explore interesting or unexpected concepts or issues raised by respondents Opportunity for deep probing of issues Opportunity for deep probing of issues May reveal previously unknown personal issues or experiences May reveal previously unknown personal issues or experiences Quotes have impact! Quotes have impact! Source: Patton, Michael Q. Qualitative Research and Evaluation Methods. 3 rd ed. (Thousand Oaks, CA: Sage, 2002) 10 of 20

11. Observation Purpose A method enabling researchers to systematically learn about and document the activities of respondents in their natural, organizational setting by observing and/or participating in those activities. Kawulich, Barbara B. “Participant Observation as a Data Collection Method.” Forum: Qualitative Social Research 6 (May 2005): Article 43 [http://www.qualitative-research.net/fqs-texte/2-05/05-2-43-e.htm]; Lofland, John and Lyn H. Lofland. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. 3 rd ed. Belmont, CA: Wadsworth, 1995. 11 of 20

12. Observation Design Considerations General Forms Participatory Observation Participatory Observation Passive Observation of Interactions Passive Observation of Interactions Covert Observation of Interactions Covert Observation of Interactions Inobtrusive Observation concerning the Residues and Consequences of Interactions Inobtrusive Observation concerning the Residues and Consequences of Interactions Observation is useful :  When you need direct information about phenomena  When you are attempting to understand an ongoing behavior, process, event, or situation as it unfolds in real time  When there is physical evidence or outcomes that can be readily seen  When written or discursive forms of data collection procedures appear to be inadequate or inappropriate 12 of 20

13. Observation Outcomes for Assessment Can facilitate multidimensional and long-term imbeddedness in the “natural setting” Can facilitate multidimensional and long-term imbeddedness in the “natural setting” Melds together aspects of looking and listening and watching and asking (connect what people are saying to what they are actually doing) Melds together aspects of looking and listening and watching and asking (connect what people are saying to what they are actually doing) Unmasks everyday interaction between people and organizations; reveals the mundane Unmasks everyday interaction between people and organizations; reveals the mundane Cases can be revisited repeatedly Cases can be revisited repeatedly 13 of 20

14. Qualitative Case Studies Purpose May be considered a unit of analysis or a choice of what is to be studied; the case is a single bounded system that can encompass time (development or history of a program), space (the object of study is located in a particular place), and/or groupings of components (number of participants; types of programs). Stake, Robert E. The Art of Case Study Research. Thousand Oaks, CA: Sage, 1995; Case Study Evaluations. Washington, DC: United States General Accounting Office, Program Evaluation and Methodology Division, 1990. 14 of 20

15. Case Studies Design Considerations Cases can be:  Individuals—comparative sample of individual experiences  Groups—sample of individuals who share a common identity  Communities—sample of individuals with ethnic, cultural, spiritual or other types of characteristics in common  Programs—comparative sample of projects or services intended to meet a public need  Policies—comparative sample of plans of action  Organizations—comparative sample of a formal group of people with one or more shared goals  Events—sample of noteworthy occurrences that trigger an organizational response or state of transition 15 of 20

16. Qualitative Case Studies Applications for Assessment Illustrative: Descriptive and intended to add “realism” and in-depth examples to other information about a program or policy. Illustrative: Descriptive and intended to add “realism” and in-depth examples to other information about a program or policy. Exploratory: Also descriptive, but aimed at generating hypotheses for later investigation rather than for purposes of illustrating. Exploratory: Also descriptive, but aimed at generating hypotheses for later investigation rather than for purposes of illustrating. Deviant/Critical Instance: Examines a single instance of unique interest or serves as a critical test of an assertion about a program, problem, or strategy. Deviant/Critical Instance: Examines a single instance of unique interest or serves as a critical test of an assertion about a program, problem, or strategy. Program Implementation: Investigates operations, often at several sites, and with a subjective, value laden, and/or affective emphasis. Program Implementation: Investigates operations, often at several sites, and with a subjective, value laden, and/or affective emphasis. Program Effects: This application uses the case study to examine causality and usually involves multi-site, multi-method assessments. Program Effects: This application uses the case study to examine causality and usually involves multi-site, multi-method assessments. Cumulative: This brings together findings from many case studies to answer an evaluation question, whether descriptive, normative, or cause- and-effect. Cumulative: This brings together findings from many case studies to answer an evaluation question, whether descriptive, normative, or cause- and-effect. 16 of 20

17. Other Techniques for Assessment Student Portfolios A form of individual performance assessment “in which students’ work is systematically collected and carefully reviewed for evidence of learning and development” [Palomba and Banta, 1999, p. 131]. Content Analysis A set of techniques used for the systematic analysis of communicative content [e.g., text, discourse, images, etc.] Life Histories A practice of collecting oral testimonies intended to “advance understanding about the complex interactions between individuals’ lives and the institutional and societal contexts in which they are lived” [Cole and Knowles, 2001, p. 126] 17 of 20

18. Some Benefits Provides deep meaning Provides deep meaning Brings contextual clarity to lived experiences Enhances subjective awareness of others Reveals the hidden or mundane Aims are not pre-determined Provides a multi- dimensional and dynamic picture of group experience Provides a multi- dimensional and dynamic picture of group experience Helps determine questions and types of follow-up research Helps determine questions and types of follow-up research Some Pitfalls Institutional Review Boards Institutional Review Boards Quantity of information Quantity of information Time and energy Time and energy Decision-makers unfamiliar with qualitative methods and their application to problem-solving Decision-makers unfamiliar with qualitative methods and their application to problem-solving Ethical issues Ethical issues Difficult to separate phenomena into distinct and workable units of analysis Difficult to separate phenomena into distinct and workable units of analysis Inquiries do not lend themselves to replicability or generalizability Inquiries do not lend themselves to replicability or generalizability

19. Selected Resources Creswell, John W. Qualitative Inquiry and Research Design: Choosing Among Five Traditions. Thousand Oaks, CA: Sage Publications, 1998. Creswell, John W. Qualitative Inquiry and Research Design: Choosing Among Five Traditions. Thousand Oaks, CA: Sage Publications, 1998. Denzin, Norman K. and Yvonna S. Lincoln, eds. Handbook of Qualitative Research. 1 st edition. Thousand Oaks, CA: Sage Publications, 1994. Denzin, Norman K. and Yvonna S. Lincoln, eds. Handbook of Qualitative Research. 1 st edition. Thousand Oaks, CA: Sage Publications, 1994. Hatch, J. Amos. Doing Qualitative Research in Educational Settings. Albany, NY: State University of New York Press, 2002. Hatch, J. Amos. Doing Qualitative Research in Educational Settings. Albany, NY: State University of New York Press, 2002. Kvale, Steiner. InterViews: An Introduction to Qualitative Research Interviews. Thousand Oaks, CA: Sage Publications, 1996. Kvale, Steiner. InterViews: An Introduction to Qualitative Research Interviews. Thousand Oaks, CA: Sage Publications, 1996. Lofland, John and Lyn H. Lofland. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Belmont, CA: Wadsworth, 1995. Lofland, John and Lyn H. Lofland. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Belmont, CA: Wadsworth, 1995. Patton, Michael Q. Qualitative Research and Evaluation Methods. 3 rd edition. Thousand Oaks, CA: Sage Publications, 2002. Patton, Michael Q. Qualitative Research and Evaluation Methods. 3 rd edition. Thousand Oaks, CA: Sage Publications, 2002. Stake, Robert E. The Art of Case Study Research. Thousand Oaks, CA: Sage Publications, 1995 Stake, Robert E. The Art of Case Study Research. Thousand Oaks, CA: Sage Publications, 1995 Stern, Elliot, ed. Evaluation Research Methods. 4 vols. Thousand Oaks, CA: Sage Publications, 2005. Stern, Elliot, ed. Evaluation Research Methods. 4 vols. Thousand Oaks, CA: Sage Publications, 2005.

20. Contact Information Dr. Robert V. Labaree Applied Social Sciences Librarian Von KleinSmid Center Library University of Southern California Los Angeles, CA 90089-0048 213-740-5824labaree@usc.edu 20 of 20