1. QUALITATIVE METHODS IN PROGRAM EVALUATION Brian J. English Ph.D. International Public Policy and Management Program University of Southern California

2. “Discovery consists in seeing what everyone else has seen and thinking what no one else has thought.” Albert Szent-Gyorgi, 1937 Nobel Prize Physiology and Medicine

3. Module Objectives What is qualitative research? What is qualitative research? How do quantitative research and qualitative research differ? How do quantitative research and qualitative research differ? What qualitative methods are appropriate for program evaluation? What qualitative methods are appropriate for program evaluation? Why are qualitative methods appropriate for gathering data to evaluate public policy and intervention programs? Why are qualitative methods appropriate for gathering data to evaluate public policy and intervention programs?

4. Module Objectives How can qualitative and quantitative methods complement each other? How can qualitative and quantitative methods complement each other? How can qualitative data aid in the planning and implementation of intervention programs? How can qualitative data aid in the planning and implementation of intervention programs? NOTE: This module will not cover data coding or the analysis of qualitative data.

5. Defining Qualitative Research “A particular tradition in social science that fundamentally depends on watching people in their own territory and interacting with them in their own language, on their own terms.” (Kirk and Miller, 1986, p.9)

6. Defining Qualitative Research "Qualitative research is a loosely defined category of research designs or models, all of which elicit verbal, visual, tactile, olfactory, and gustatory data in the form of descriptive narratives like field notes, recordings, or other transcriptions from audio- and videotapes and other written records and pictures or films.“ Judith Preissle

7. Brian’s Definitions Interpretive Interpretive Naturalistic Naturalistic Can be Ethnographic (a specific type) Can be Ethnographic (a specific type) Descriptive Descriptive Uses all 5 (or 6) of the senses Uses all 5 (or 6) of the senses Writers’ research Writers’ research

11. How do Quantitative and Qualitative Research differ? Qualitative researchers try to understand reality as it is construed by the persons being studied. Qualitative researchers try to understand reality as it is construed by the persons being studied. Qualitative researchers try to understand the meaning of particular activities or beliefs in the context of the culture being considered Qualitative researchers try to understand the meaning of particular activities or beliefs in the context of the culture being considered (Valadez and Bamberger, 1994)

12. Qualitative Inductive Inductive Subjective/insider centered Subjective/insider centered Process oriented Process oriented Lack of control of variables Lack of control of variablesQuantitative Deductive Objective/outsider centered Outcome oriented Attempt to control variables

13. Qualitative Goal: Understand actors view Goal: Understand actors view Discovery Oriented Discovery Oriented Explanatory ExplanatoryQuantitative Goal: Find facts and causes Verification Oriented Confirmatory

14. Differences in Data

15. What Qualitative Research Methods are appropriate for gathering data for Public Policy Program Evaluation? Document analysis of archival records Document analysis of archival records Observation & Participant Observation Observation & Participant Observation Interviews/Focus groups Interviews/Focus groups Chismoso & Laquachero Chismoso & Laquachero Triangulation Triangulation

16. Document Analysis  Newspapers  Birth/death records  Environmental Impact Reports  Financial Reports  Previous Evaluations  Journals  Film & Photos (Grand Canyon example)

17. “You can observe a lot just by watching." --Yogi Berra

18. Observation  Gaining Access  Participant Observation  Etic vs. Emic  Shadowing

19. Interviews/Focus Groups  Key Informants  Confidential Informants  Informal Interviews  Structured Interviews

20. Chismoso & Laquachero  Talking and Walking  Need an Outgoing, Communicative Approach  Gaining trust  Note-taking & recordings can frighten or turn-off potential informants

21. Triangulation Document Analysis Interviews Observation

22. Why are qualitative methods appropriate for gathering data to evaluate public policy and intervention programs? & How can qualitative and quantitative methods complement each other?

23. The intricate relationship between human behavior and public policies can restrict or promote specific behaviors. Examples:  Restrictions on fishing  Prescription Drug policies  Zoning Legislation

24. The connections between public policy, infrastructure planning, environmental degradation, health, education and poverty  East Coast Land Developers  Waste Management and Dengue Fever  Coastal Resource Management

25. Numbers do not always reflect reality  Ivy tower syndrome  Glass ceiling syndrome  US prisons  Money drives policy; Money drives research

26. The data collected in qualitative studies can be useful in developing quantitative studies.  Determining Indicators  Identifying factors/variables  Adding Perspective

27. The learning process and behavioral changes that take place during a program intervention may best be explored in a natural setting over a long period of time.  Crosscheck information gathered in quantitative surveys  Aid in planning educational activities  Aid in Program revision

28. How can data aid in the planning and implementation of intervention programs?  Qualitative methods can reveal how the local population feels about a new project and how that project may affect their lives.  Program designers have a moral obligation to try to understand how locals feel about interventions.

29. Strengths Depth & Detail Depth & Detail Openness Openness Insider Perspective Insider Perspective Tries to avoid pre-judging Tries to avoid pre-judging Can be less expensive Can be less expensiveWeaknesses Smaller Sample Less easily generalized Difficult to make systematic comparisons Dependant on researchers skills The effect of the researcher’s participation

30. Considerations for Further Thought Unobtrusiveness Unobtrusiveness Fitting in Fitting in Ethical Issues Ethical Issues Legal Issues Legal Issues Coding Coding Analysis Analysis Drawing Inferences Drawing Inferences Validity Validity

31. Comments & Questions