1. Using Qualitative Comparative Analysis to Identify Evidence-based Best Practices in Program Evaluation Brandy Farrar 1 and Jennifer Craft Morgan 2 1 American Institutes for Research 2 Georgia State University

3. Background Challenge: Evaluating programs that have component programs that are non-standard or varied in their approach to leveraging key program outcomes. Strategy: Comparative case study or process evaluation.

4. Background Limitations of traditional approaches: 1.Difficult to identify which programmatic conditions (or combinations of conditions) have influence on program outcomes. 2.The likelihood that multiple co-occurring programmatic conditions lead to a desired outcome complicates attempts to identify evidence-based best practices.

5. Objective The purpose of this presentation is to illustrate the utility of using Qualitative Comparative Analysis in program evaluations involving mixed-method case study data.

7. Purpose: Create education and career advancement for frontline health care workers Partnerships Health care employers Educational Institutions Community Organizations

8. Sample program description Medical Assistant Medical Terminology Medical Interpreting I Medical Interpreting II Job Shadowing Certified Auxiliary Interpreter 9 hours college credit Wage premium

9. Data and Methods  Case study methodology  Process and outcome evaluation  Mixed methods (e.g. surveys, interviews)  Longitudinal  Formative and summative reports to sites and funder

10. Data Table 1 Jobs to Careers Evaluation Data Elements Type # of data elements# of people Semi-structured interviews with key informants 467241 Focus groups with frontline workers39282 Focus groups with frontline supervisors33184 Organizational surveys100N/A Implementation assessment tool17N/A Frontline worker survey - baselineN/A1129 Frontline worker survey - follow upN/A576 Local economic indicators17N/A

11. Qualitative Comparative Analysis  Allows researchers to quantify and empirically test qualitative data without losing its richness and substantive complexity  Allows evaluators to assess significance, magnitude, and parsimony of qualitatively assessed concepts

12. Qualitative Comparative Analysis 1.Identify the causal conditions that are likely to be related to the outcome of interest Condition Career self-efficacy Barrier reduction Career guidance Experiential learning Socio-emotional support Positive role models Organizational learning culture Community context

13. Qualitative Comparative Analysis 2. Calibration Condition IndicatorDescriptionCalibration metric Barrier reduction Educational release time Paid work time to attend class or complete coursework 1 = Significant paid release time that includes study time and was widely granted by supervisors Experiential learning N/A Extent to which experiential learning was used in curriculum delivery. Examples of experiential learning include: discussions of real clients in class; class assignments that have to be performed within the context of work responsibilities; credit for prior learning/work experience; precepting; job shadowing, etc. 1 = Experiential learning occurred more often than traditional delivery methods such as lecture Organizational learning culture N/AAggregated mean based on a scale of 11 survey items (see Appendix I) 1 = Aggregate mean greater than or equal to 3.0

14. Qualitative Comparative Analysis Sample Raw data table casebarrierguidanceexperientialsupportcohortorgculturecommunityefficacy A10111001 B01010010 C01001101 D01111101 E10010101 F10110000 G00110000 H00010010 I00011100 J10011000 K11101000 L01010000 M11111100 N01111001 O10111000 P10110101 Q10111100 R11110001

15. Qualitative Comparative Analysis 3. Analysis - Tests of consistency and coverage  If a particular combination of conditions is present when the outcome of interest is also present in the vast majority of cases displaying this set of conditions (80% or more of the time), consistency is high  If a particular set of conditions is one of a few, versus one of many sets of conditions that are present when the outcome is also present, coverage is high

16. Qualitative Comparative Analysis 3. Analysis – Tests of consistency and coverage Consistency Coverage Case Outcome Combinations * Blue combination has best consistency * VS. Better coverage for blue combination

17. Qualitative Comparative Analysis 4. Analysis – Parsimony  Necessary versus sufficient combinations of conditions associated with the outcome  A condition is necessary if it is present in all instances of the outcome.  A condition is sufficient if it can produce an outcome, but is not the only condition that can produce the outcome.

19. Combinations of causal conditions associated with high career self- efficacy  No necessary conditions  4 combinations of sufficient conditions

20. Combinations of causal conditions associated with high career self- efficacy

22. Discussion Need a comprehensive approach to engendering career self-efficacy But one size does not fit all Community context was not as important as we suspected Focused efforts in a few, rather than all, areas can still produce high career self-efficacy among program participants

23. Conclusion  QCA allows for a more precise specification regarding:  Importance and relevance of qualitatively assessed process evaluation constructs  How constructs work together to facilitate or constrain desired program outcomes  What conditions are necessary and which are sufficient helping resource-strapped program planners to develop effective, lower resourced programs

24. Questions? For further information contact: Brandy Farrar bfarrar@air.org (202) 403 - 5416