1. Session 2: Understanding Program Theory and Implementation Theory

2. Objectives for Today Formative v. Summative Evaluations Program Theory v. Implementation Theory Example: STOP-DWI Stages of Formative Evaluation Assign Groups for Focus Group/Survey Project

3. Formative Evaluation What is being implemented? –Program description –Stages? Steps? Services? Procedures? Are the necessary steps in service delivery being adequately coordinated and executed? –Consistent with expectations or guidelines? –How do clients experience the program?

4. Summative Evaluation How well did the program achieve its intended impact? Are there any unintended or unforeseen consequences? Identify changes in conditions, behavior, or status that are directly related to the program

5. Formative or Summative? Number of visits to food pantry Satisfaction with job referral system Lower rates of heart disease Response time of operators to hotline callers Cleaner drinking water Citizen involvement in decision-making Sustained employment for 6 months Lower infant mortality rates

6. Program v. Implementation Theory Program Theory = mechanisms of change –“specifies a chain of causal assumptions linking program resources, activities, intermediate outcomes, and ultimate goals” –“the mechanisms that mediate between service delivery (and receipt) of the program and the emergence of the outcomes of interest” Implementation Theory = program activities –Translation of program objectives into services and an operating program

8. STOP-DWI in NY State In 1981, NY State passed STOP-DWI law to reduce drunken-driving and alcohol-related accidents. What is the implementation theory? –What activities or services comprise the STOP-DWI program? What is the program theory? –How will the program affect behavioral change? How would we know if it was being implemented well? What might affect successful implementation of STOP-DWI programs?

9. Outcomes of STOP-DWI in NY Revenues from DWI fines: –1982 fines = $156,313; 2000 fines = $925,666 36 alcohol-related fatalities in 1981, 2 alcohol-related fatalities in 2000 (–94%) -67 fatalities total in 1981, 46 in 2000 (-31%) DWI arrests in Onondaga County: –1980 = 1,069; 1990 = 2,191; 2000 = 1,461 Is STOP-DWI a success?

10. Outcomes of STOP-DWI in NY How do we know if the program is having an impact? Program v. non-program effects What factors affect the number of accident-related fatalities? What factors other than the program might affect drunk-driving rates?

11. Stages in Developing an Evaluation Stage One –Defining the Evaluation –Program goals, implementation theory, type of evaluation, stakeholders Stage Two - Develop Analysis Strategy –Tracking implementation and outcomes –Proper data for the question(s) of interest Stage Three - Implementation of Data Collection Stage Four - Analysis, Report Writing, and Presentation

13. Readings for Next Time: Trochim, Chapters 1 – 3 Also, in-Class Data Activity next Monday