QUALITATIVE EVALUATION OVERVIEW Kim Koester Director of Qualitative Research AIDS Policy Research Center Center for AIDS Prevention Studies UCSF

Objectives  To teach you to understand the application of and methods associated with qualitative evaluation  To provide an overview a peek “under the hood” of the qualitative analytic process.

Outline of talk 1. Explain qualitative research process 1. Methods 2. Procedures 3. Sampling 4. Analysis 2. Demystify the analytic activities 1. Briefly describe different ways to approach qualitative data analysis. 3. Dig in to practical issues 1. Talk about resources required 2. Touch on issues that must be addressed in writing up data for publication

Questions for you  What comes to mind when you think of qualitative methods?  Who plans to incorporate qualitative methods into your evaluation plan for this class?  Has anyone participated in a qualitative research project, as either an informant or as a member of a research team?

What is Qualitative Research? Qualitative research is an approach to scientific inquiry that relies on more naturalistic, humanistic and interactive processes. The methods are primarily language based, with data in the form of words rather than numbers. Joan S. Ash and Kenneth P. Guappone 2007

Qualitative Inquiry Areas of Emphasis  Emphasizes a holistic perspective  Strive to understand a program and situation as a whole  Emphasizes the importance of understanding the meanings of human behavior and the social-cultural context of social interaction.  Emphasizes the emic perspective

Qualitative researchers empathize and identify with the people they study in order to understand how those people see things. Taylor and Bogdan

Qualitative Methods  Geared towards exploration, discovery, inductive logic  Defined in comparison to deductive research that begins with an hypothesis.

Comparing Qualitative & Quantitative Methods  Inductive  Small n  Purposeful sample  Open-ended questions, geared towards discovery  Seeking holistic understanding  Not generalizable  Investigates how or why  Deductive  Large n  Random sample  Closed-ended questions, predetermined measures  Individual variables  Generalizable  Investigates how many when, where

Qualitative Techniques  In-depth interviewing (IDI)  Key informant interviews  Semi-structured interviews  Narrative interviews Life history interviews Sexual history interviews Critical incident interviews  Focus groups or Discussion groups  Ethnography  Participant observation – shadowing/tag alongs

Selecting the method  The method you select depends entirely on the research question. Give me an example and we’ll talk through the best method.  What methods used in the articles you read for today?

Sampling  Sample size in qualitative research vary from project to project; “there are no rules for sample size”  Purposive or purposeful sampling is the most common strategy in applied research/evaluation.  Purposeful sampling relies on the selection of information- rich cases; information rich cases allow learning a great deal about the issues that are of central importance to the evaluation Patton delineated 16 different types of sampling frames

Designing Interview Guides  The purpose of the interview guide is to list the questions or areas of inquiry to be approached in the interview setting.  Semi-structured interviews  In-depth interviews

 Example Interview Guide DOMAIN: PERSONAL HISTORY/LIFE CIRCUMSTANCES INTENT: The intent of this section is to learn about what has happened since the person was released and about how they are doing, in general. QUESTIONS: Tell me how you have been meeting your day-to-day survival needs, for example, housing, money, food, etc.? Tell me about your experiences re-connecting with friends and family since you got out of prison.

Qualitative Analysis  Qualitative analysis is the process by which you turn your data into findings.  What are you doing when you are analyzing data?  Reducing the sheer volume of raw data  Separating out trivia from significance  Identifying significant patterns

Qualitative Analysis  Driven by goals and aims of the evaluation  Describe program  Elucidate program dynamics  Identify patterns  Confirm or disconfirm quantitative findings

Key analytic activities  Thinking  Reading  Noticing  Reacting  Learning  Producing insights  Reading  Sharing & discussing Refining insights Reading Labeling Producing insights Labeling Reading Parsing Assembling Understanding Interpreting Reading And so on

Phenomenology  What is the meaning, structure, and essence of the lived experience of this phenomenon for this person or group of people?  Goal: to elucidate meaning of lived experience

Narrative Analysis  What does this narrative or story reveal about the person and world from which it came? How can this narrative be interpreted so that it provides an understanding of and illuminates the life and culture that created it?  Comparing stories and cases to create meaning  Goal: to interpret stories to reveal cultural patterns

Grounded Theory  What theory emerges from a systematic comparative analysis and is grounded in fieldwork so as to explain what has been and is observed?  Theoretical sampling  Theoretical saturation  Open and axial coding Goal: to generate explanatory theory from social processes Best uses: unstructured in-depth interviews, structured data sets with rich, lengthy narratives

Framework Analysis  Commonly used in applied policy research, this approach involves a set of distinct, yet highly interconnected analytic stages. Familiarization w/data Identify thematic framework Indexing or coding Charting Mapping and Interpretation  Goal: to follow a transparent analytic process  Best uses: interdisciplinary, team-based research, in depth interviews and focus group data

Content Analysis Involves tagging a set of texts or other artifacts with codes that are derived from theory or from prior knowledge and then analyzing the distribution of the codes, usually statistically Once codes are developed, they become the unit of analysis rather than the utterances to which codes are assigned Goal: begin with deductive application of a priori codes, but remain open to inductive revelations. Best uses: Structured in-depth interviews, focus group

Software  Qualitative software packages exist to facilitate the organization of qualitative data.  Dedoose single user $(15 per month)  Atlas.ti single user $1800  Multi-user (5) - $6700  Nvivo single user - $1300  Multi-user (5) - $5200

Simply observing and interviewing do not ensure that the research is qualitative; the qualitative researcher must also interpret the beliefs and behaviors of participants. Valerie Janesick

Disposition Ideas Characteristics of analysts:  Significant amount of patience and tolerance for reading and re-reading data  Able to sit with ambiguity  Non-linear orientation  Creative