CRJS 4466 PROGRAM & POLICY EVALUATION LECTURE #5 Evaluation projects Questions?

Qualitative Evaluation Methods

Qualitative Methods – strengths Carol Weiss (1998) – the coming of age of qualitative methods note the Campbell versus Cronbach controversy here qualitative methods are an important technique for under- standing the ‘context’ of success or failure of programs qualitative methods most commonly used in process evaluation (formative) or the ‘how’ and ‘why’ of a program and its workings ability to represent subtleties and complexities of program functioning and dynamics

Qualitative Methods - strengths many evaluators like the interpersonal nature of qualitative methodologies qualitative approaches are often ‘action oriented’ a choice of ‘depth’ over ‘breadth’

Qualitative Methods - limitations reliability and validity of measures interviewer bias anecdotal labour-intensive costly less useful for reporting on program outcomes

Qualitative Methods – Single System Evaluations evaluating an intervention with a single client system, usually quantitatively during the course of treatment limited or questionable usefulness of qualitative methodology

4. Qualitative Methods – Focused Qualitative Evaluation (FQE) reflects the immediacy of much evaluation – not time for extended research, ethnomethodological development use of FG and rapid ethnographic assessment

Qualitative Methods – Mixed Methods ‘triangulation’, combining both quantitative and qualitative methods ‘temporal sequencing’ and qualitative subsamples benefit of both breadth and depth

Qualitative Methods – Action (Participatory) Research overtly ‘action-oriented’ approach primarily among those conducting qualitative evaluations participatory, collaborative, empowering – now an expectation in some evaluation projects ‘praxis’ orientation

Qualitative Methods – Research Design units of analysis and comparative designs gaining access, key informants, and sponsorship reciprocity, payback and feedback – and the methodological, ethical concerns here (‘false hope’) sampling considerations deviant case sampling/typical case sampling maximum variation sampling snowball sampling purposive sampling convenience sampling random sampling

Qualitative Methods – Research Design data collection strategies on-site observation participant observation reflexivity use of interview guide use of focus groups cross-validation among interviewers use of documents when to stop data collection – the issue of saturation

Qualitative Methods – Data Management and Analysis use of qualitative data analysis programs: Ethnograph, AskSam QSR (Nudist) coding data emic (indigenous) and etic (researcher-created) coding quality control mechanisms reporting – issues related to qualitative methodologies

Table 5-1:Key Differences Between Qualitative and Quantitative Evaluation Approaches

Table 5.2:Summary of Key Questions in Conducting Qualitative Evaluation Assessments and Evaluation Studies

Table 5-3:Sampling Strategies for Qualitative Evaluations

Table 5-4:Open Ended Questions for the Economic Development Agreement Stakeholders Project

Table 5-5:Some Basics of Face-to-Face Interviewing

Table 5-6:Helpful Hints as You Analyze Qualitative Data

Table 5-7:Comparing Qualitative Validity with Experimental and Quasi-Experimental Validity

Table 5-8:Ways of Testing and Confirming Qualitative Findings

Focus Group Question: “Based on your own experiences of the Criminal Justice Program here at Nipissing University, what do you like about the program? What don’t you like about the program?