1. Presented by CSAP’s CAPT Southwest Regional Team Southwest Prevention Center The University of Oklahoma 1639 Cross Center Drive, Suite 254 Norman, OK 73019 (800) 853-2572

2. Approaches to Prevention Evaluation How Do You Feel About Evaluation? –What comes up for you when you think about it?

3. Learning Objectives Understand the purposes of evaluation Understand the key components of evaluation Use a logic model as a guide to create an evaluation plan Identify measures and sources of data for evaluation Identify the benefits of internal and external evaluation Use evaluation findings to assist with decision- making

4. Prevention Evaluation Why is it important?

5. Evidence-Based Prevention A prevention activity is judged to be evidence-based if “good” evaluation – research that has been shown to be rigorous according to a set of carefully defined criteria – demonstrates that the activity is effective. The evaluation should demonstrate that: the activity produces the expected positive results intended, and these results can be attributed to the activity or program rather than to other factors.

6. Evaluation The systematic collection and analysis of information about program activities, characteristics, and outcomes to reduce uncertainty, improve effectiveness, and make decisions.

7. It’s About Utility Planning programs Monitoring implementation of programs Improving or redesigning programs Advancing knowledge of innovative programs Providing evidence of program effectiveness

8. Definitions Process Evaluation – Documenting program implementation Outcome Evaluation – Documenting effects that you expect to achieve after the program is implemented

9. Traditional vs. Collaborative Evaluation 9 TraditionalCollaborative Done to the programDone with the program Evaluator operates apart from the program Evaluator operates in concert with the program Evaluator decidesEvaluator advises Evaluator retrieves information from program staff as needed to plan and carry out the study Program staff are participants in planning and carrying out the study Evaluator interacts relatively infrequently through the program director Evaluation interacts regularly through the program staff and other stakeholders

10. The Collaborative Model The primary mechanism is an evaluation team made up of: Evaluator Program Staff Other Stakeholders (e.g., in a school-based program, stakeholders may include curriculum designers, school board members, teachers, parents, students) 10

11. Framework for Evaluation 11 Describe the Program Focus the EvaluationDesign SelectAppropriateMethods JustifyConclusions Ensure Use and Share Lessons Learned EngageStakeholders

12. Step 1: Engage Stakeholders 12

13. Stakeholders Those organizations and individuals who care about either the program or the evaluation findings In general, anyone who has something to gain or lose from the program 13

14. Activity 1 : Who Are Your Stakeholders? Identify the stakeholders in your program Identify their interests Rank order from most to least important –Program stakeholders –Evaluation stakeholders 14

15. Activity 1 :Things to Keep in Mind Did you put yourself on the list? Did you identify competing needs? Were the agendas of all stakeholders explicit? Were you clear about what could and couldn’t be accomplished? 15

16. Step 2: Describe the Program 16

17. Definition of a Logic Model Description of what a program is expected to achieve and how it is expected to work A map linking together a project’s goals, activities, services, and inclusive of assumptions 17

18. Benefits of a Logic Model 1.Develops understanding 2.Helps monitor progress 3.Serves as an evaluation framework 4.Helps expose assumptions 5.Helps restrain over-promising 6.Promotes communications 18

19. Activity 2: Let Purpose Be Your Guide Discuss among yourselves what’s the purpose of your evaluation; the utility. Acquaint your S2S evaluator consultant and/or your invited evaluators with your program and what purpose you hope that an evaluation would help with. 19

20. 20 Blank Logic Model GOALS FOCUS POPULATION STRATEGIES“IF-THEN” STATEMENTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES A. In order to address the level of this risk or protective factor: B. For these people: C. We will do the following program activities/strategies (what, where, and how much): D. We expect that this activity will lead to changes in these risk/protective factors, which in turn will lead to our program goal: E. We will know these changes have occurred if: F. We will know we are reaching our goals if: 1. Logic Model: 2. Evaluation Questions: 3. Data Collection Sources:

21. Designing a Logic Model A.Goals: What risk and protective factors will be addressed? B.Focus Population: Who will participate in, or be influenced by, the program? C.Strategies: What services and activities will be provided? D.“If-Then” Statements: How will these activities lead to expected outcomes? E.Short-Term Outcomes: What immediate changes are expected for individuals, organizations, or communities? F.Long-Term Outcomes: What changes would the program ultimately like to create? 21

22. 22 Sample Logic Model A GOALS FOCUS POPULATION STRATEGIES“IF-THEN” STATEMENTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES A. In order to address the level of this risk or protective factor: B. For these people: C. We will do the following program activities/strategies (what, where, and how much): D. We expect that this activity will lead to changes in these risk/protective factors, which in turn will lead to our program goal: E. We will know these changes have occurred if: F. We will know we are reaching our goals if: 1. Logic Model: Reduction in Academic Failure

23. 23 Sample Logic Model A GOALS FOCUS POPULATION STRATEGIES“IF-THEN” STATEMENTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES A. In order to address the level of this risk or protective factor: B. For these people: C. We will do the following program activities/strategies (what, where, and how much): D. We expect that this activity will lead to changes in these risk/protective factors, which in turn will lead to our program goal: E. We will know these changes have occurred if: F. We will know we are reaching our goals if: 1. Logic Model: Reduction in Academic Failure Children in grades 1-3 at the local elementary school who are struggling academically as identified by teachers

24. 24 Sample Logic Model A GOALS FOCUS POPULATION STRATEGIES“IF-THEN” STATEMENTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES A. In order to address the level of this risk or protective factor: B. For these people: C. We will do the following program activities/strategies (what, where, and how much): D. We expect that this activity will lead to changes in these risk/protective factors, which in turn will lead to our program goal: E. We will know these changes have occurred if: F. We will know we are reaching our goals if: 1. Logic Model: Reduction in Academic Failure Children in grades 1-3 at the local elementary school who are struggling academically as identified by teachers Academic Tutoring: 3 hours per week for one school year 50 students

25. Assumptions Identify the assumptions underlying your program. Do your program activities lead logically to your goals? How and why do you expect your program to achieve your goals? What are the steps that will lead logically from your program activities to your goals? 25

26. “If-Then” Statements IF the program invests time and money to develop an inventory of the drug-free activities…THEN youth will be more informed about opportunities within the community. IF youth know what’s available…THEN they’ll be more likely to participate. IF youth participate in alternative drug-free activities…THEN they’ll be more likely to develop friendships with non-using peers and THEN be less likely to use ATOD themselves. 26

27. 27 Sample Logic Model A GOALS FOCUS POPULATION STRATEGIES“IF-THEN” STATEMENTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES A. In order to address the level of this risk or protective factor: B. For these people: C. We will do the following program activities/strategies (what, where, and how much): D. We expect that this activity will lead to changes in these risk/protective factors, which in turn will lead to our program goal: E. We will know these changes have occurred if: F. We will know we are reaching our goals if: 1. Logic Model: Reduction in Academic Failure Children in grades 1-3 at the local elementary school who are struggling academically as identified by teachers Academic Tutoring: 3 hours per week for one school year 50 students *If tutoring is offered to students having academic problems, then students will have the opportunity to improve their academic skills. *If identified students take the opportunity, they will improve their academic skills. *If they improve their academic skills, they will not fail in school. *If they don’t fail in school, they will be less likely to abuse alcohol, tobacco, and other drugs.

28. The Short & Long of It Short-Term Outcomes – the immediate program effects that you expect to achieve (e.g., improving problem solving skills) Long-Term Outcomes – the long-term or ultimate effects of the program (e.g., reducing drug use) 28

29. A Word about Outcomes There are no right number of outcomes The more immediate the outcome, the more influence the program has over its achievement The longer term the outcome, the less direct influence a program has over its achievement Because other forces affect an outcome doesn’t mean it shouldn’t be included Long-term shouldn’t go beyond the program’s purpose or focus audience 29

30. 30 Sample Logic Model A GOALS FOCUS POPULATION STRATEGIES“IF-THEN” STATEMENTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES A. In order to address the level of this risk or protective factor: B. For these people: C. We will do the following program activities/strategies (what, where, and how much): D. We expect that this activity will lead to changes in these risk/protective factors, which in turn will lead to our program goal: E. We will know these changes have occurred if: F. We will know we are reaching our goals if: 1. Logic Model: Reduction in Academic Failure Children in grades 1-3 at the local elementary school who are struggling academically as identified by teachers Academic Tutoring: 3 hours per week for one school year 50 students *If tutoring is offered to students having academic problems, then students will have the opportunity to improve their academic skills. *If identified students take the opportunity, they will improve their academic skills. *If they improve their academic skills, they will not fail in school. *If they don’t fail in school, they will be less likely to abuse alcohol, tobacco, and other drugs. Participants grades improve. Participants move to next grade level on time. Participants do not begin using ATOD within five years of participating in the program. (Participants will be in 6-8 grades)

31. Activity 3: Generating a Logic Model Review the example, “Sample Logic Model A.” Complete Row 1 of the Blank Logic Model work sheet using your program. Record your Logic Model on chart paper and post. 31

32. Step 3: Focus the Evaluation Design 32

33. Activity 4: Logic Model Focus Focus your Logic Model on the area in which you would like to direct your evaluation efforts. You may have brought a Program Logic Model (or you may need to develop one), review and assess your possibilities recalling the purpose you may have identified earlier. Discuss this at your table. 33

34. Designing an Evaluation Clarify PURPOSE of evaluation that leads to  QUESTIONS that require  INFORMATION and data obtained from  METHODS 34

35. Points to Consider Keep in mind the purpose of the evaluation What’s going to be evaluated Who wants to know what When you need the information What you intend to do with the evaluation results Resources you have available for the evaluation (e.g., time, money, people) 35

36. Be realistic... But think creatively. 36

37. Tips for Generating Evaluation Questions 3 to 5 questions are often adequate. Use open-ended questions, not “yes-no.” Avoid compound questions, i.e., questions that include multiple statements. A good rule of thumb is that the questions start with “To what extent...” 37

38. Sample Evaluation Questions Process Evaluation How are resources allocated to various activities? To what extent was the program implemented as planned? What obstacles were encountered during program implementation? Outcome Evaluation Over the duration of the program, to what extent has: School attendance improved? Community-wide prevention awareness activities changed adult norms about substance use? Youth substance use decreased? 38

39. Beware of the “Black Box”! Outcome evaluations that focus solely on program effects are dangerous! 39

40. 40

41. Activity 5: Developing Your Questions Complete Row 2 of your “Blank Logic Model” work sheet, developing 2-3 evaluation questions for each column. Select the top 3 evaluation questions you would like answered. 41

42. 42 Sample Logic Model A GOALS FOCUS POPULATION STRATEGIES“IF-THEN” STATEMENTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES A. In order to address the level of this risk or protective factor: B. For these people: C. We will do the following program activities/strategies (what, where, and how much): D. We expect that this activity will lead to changes in these risk/protective factors, which in turn will lead to our program goal: E. We will know these changes have occurred if: F. We will know we are reaching our goals if: 2. Evaluation Questions: Have we achieved our goal? To what extent were academic failure rates reduced? ~Were teachers referring as intended? ~Is the method by which they refer helpful or a hindrance? ~Where their other barriers to reaching our students? ~To what extent have I implemented my tutoring program as planned? (3 hrs./wk. for 9 mos.) ~What obstacles were encountered during the process? ~Did I reach the students I intended? ~Did delivery style, setting, mentor type matter? ~Does my logic hold up? ~What assumptions have I made that may need to be challenged? ~Have participants grades improved? ~If not, why? ~Have participants moved to the next grade level? ~If not, why? ~How many improved? ~How many moved to next grade? ~Have all our students that received tutoring and moved up in grade remained drug-free? ~If not, why? ~Is there a difference in the dosage between those that didn’t start and those that did?

43. Step 4: Select Appropriate Methods 43

44. Evaluation Methods 44 QuantitativeQualitative CountingAnecdotes ChecklistsCase studies SurveysFocus groups Pre-post testsKey informant interviews Analysis of existing statisticsObservations Analysis of existing files

45. 45 Sample Logic Model A GOALS FOCUS POPULATION STRATEGIES“IF-THEN” STATEMENTS SHORT-TERM OUTCOMES LONG-TERM OUTCOMES A. In order to address the level of this risk or protective factor: B. For these people: C. We will do the following program activities/strategies (what, where, and how much): D. We expect that this activity will lead to changes in these risk/protective factors, which in turn will lead to our program goal: E. We will know these changes have occurred if: F. We will know we are reaching our goals if: 3. Data Collection Sources: ~School Advancement Records ~Program Reports ~Teachers ~Referral protocols ~Interviews ~Tutoring Aides ~Questionnaires ~Survey ~Fidelity Instrument ~Interviews ~Focus group ~Theory base ~List operating assumptions ~GPAs ~Student Achievement Tests ~Archived Advancement Records ~Student Surveys ~Dosage Logs

46. Activity 6: Identifying Data Collection Sources Complete Row 3 of your “Blank Logic Model” work sheet, identifying 2-3 data sources for each column. What gaps are you noticing? Does your logic model still link together? Are you beginning to favor a particular method? Quantitative? Qualitative? Or both? 46

47. Benefits of Quantitative Methods Standardized Succinct Easily aggregated for analysis Systematic Easily presented in short space The ability to generalize is widely accepted 47

48. Benefits of Qualitative Methods Detailed and variable Unanticipated benefits and/or concerns are possible to detect Offer explanations for short-term outcomes Help generate new ideas and/or theories 48

49. Benefits of Multi-Method Evaluation Understand program processes and outcomes from multiple perspectives Strengths of some methods compensate for weaknesses in others Results will be useful to a variety of audiences Results will be credible to a variety of audiences 49

50. Limitations of Multi-Method Evaluation Requires multiple evaluation skills and evaluation team members Cost is usually higher than single method evaluations Methodological rigor possible with single method evaluation may be sacrificed Contradictory or inconsistent findings may require additional analysis and increase complexity of reporting 50

51. Activity 7: Selecting Appropriate Methods Begin to think creatively about potential methods suitable to address some of your crafted evaluation questions. Report out on some of your ideas. What has lead you to this decision? 51

52. Step 5: Justify Conclusions 52

53. Conclusions are based on: Analysis: Isolates important findings Synthesis: Combines sources of information to reach larger understandings Interpretation: Determines meaning of the findings 53

54. Step 6: Ensure Use and Share Lessons Learned 54

55. “Use is not simply determined by some configuration of abstract factors; it is determined in large part by real, live, caring human beings.” Michael Quinn Patton, 1997 55

56. Evaluation Characteristics Report - nature and timing Relevance of findings Quality of evaluation Follow-up support and technical assistance 56

57. Consider Context Political climate Size of organization/program Competing information 57

58. Creating a Data Dissemination Plan 1.WHAT is the nature of the data you have collected? 2.WHY do you want to present the data? 3.TO WHOM are you presenting the data? 4.HOW are you going to present the data? 5.HOW MUCH data are you going to present? 6.WHO will present the data? 7.WHERE are you going to present the data? 8.WHEN are you going to present the data? 58

59. Choosing Appropriate Methods METHODS AUDIENCE Abstracts & Briefings Annual/ Evaluation Reports Fact Sheets Brochures & PostersExhibits Press Conferences Press Releases Town Meetings Current/ Potential Funder XX New Potential Funder XX Administrator XXX Board Members XXXX Community Groups XXX General Public XXXXX Organizations XX Media XXXX 59 Adapted from (1) Borden, DeBord, and Snipes and (2) Morris, Gibson, and Freeman.

60. Making Evaluation Results Useful Brief stakeholders throughout the project Help stakeholders understand data Create a dissemination plan Select the most useful media for reporting results 60

61. Framework Revisited 61 Describe the Program Focus the EvaluationDesign SelectAppropriateMethods JustifyConclusions Ensure Use and Share Lessons Learned EngageStakeholders

62. Thank You! Questions, comments, observations? Contact us: CSAP’s CAPT Southwest Regional Team swpc@ou.edu 62