University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Logic Model: A Program Performance Framework Madison Workshop December 5, 2001 Ellen Taylor-Powell, Ph.D. Evaluation Specialist UW- Extension-Cooperative Extension

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop A New Era What gets measured gets done If you don’t measure results, you can’t tell success from failure If you can’t see success, you can’t reward it If you can’t reward success, you’re probably rewarding failure

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop If you can’t see success, you can’t learn from it If you can’t recognize failure, you can’t correct it If you can demonstrate results, you can win public support Osborne and Gaebler, 1992 in MQ Patton, 1997:14

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Logic Model is… Picture of a program Graphic representation of the program “theory” or “action” – what it invests, what it does and what results Logical chain of if-then relationships; if x occurs, then y will occur Core of program planning and evaluation

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop  LOGIC Reasonable To be expected  MODEL Represents reality, isn’t reality

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Why Logic Models-- Why the Hype? Shows difference between what we do and impact we are having Provides a common vocabulary Focus on quality and continuous improvement

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Logic Model: Origins Private Sector: Total quality management Public Sector: GPRA, performance budgeting Non-Profit Sector: Outcomes measurement, eg. United Way International Arena: Results Framework of USAID, etc. Evaluators: Evaluability assessment, bennett hierarchy

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Logic Model SITUATIONSITUATION OUTCOMESINPUTSOUTPUTS

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop HEADACHEHEADACHE Feel betterGet pillsTake pills Everyday Logic Model

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop An Extension Example: Business Counseling Extension invests time and resources A variety of educational activities are provided to business owners who participate These owners gain knowledge and change practices resulting in Improved business performance

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop LOGIC MODEL: Program Performance Framework

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Logical Linkages: Series of If-Then Relationships IF THEN IF THEN IF THEN IF THEN INPUTS OUTPUT OUTCOMES Program invests time & money Resource inventory can be developed Families will know what is available Families will access services Families will have needs met

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Activity (p 2-12) Completing the causal linkages

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop LOGIC MODEL: Program Performance INPUTSOUTPUTSOUTCOMES Program investments ActivitiesParticipationShortMedium What we invest What we do Who we reach What results SO WHAT?? Long- term

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Outcomes vs. Activities BE OUTCOME DRIVEN, NOT ACTIVITY DRIVEN

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Activity (p 2-13) Program Performance Levels

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Situation Situational analysis –Need –Asset Problem analysis Priority setting Engaging others

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop INPUTS Faculty Staff Money Time Volunteers Partners Equipment Technology

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop OUTPUTS What we do Who we reach ACTIVITIES Workshops Counseling On-farm research Facilitation Product development Curriculum design Trainings Field days Conferences Media PARTICIPATION Participants Clients Customers Users

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop OUTCOMES What results for individuals, businesses, communities..… SHORT Learning Awareness Knowledge Attitudes Skills Opinion Aspirations Motivation MEDIUM Action Behavior Practice Decisions Policies Social action LONG-TERM Conditions Human Economic Civic Environment

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Staff Money Partners Design parent ed curriculum Provide 6 training session Targeted parents attend Parents increase knowledge of child dev Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect INPUTSOUTPUTS OUTCOMES

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Activity ( p 2-24 ) Which are the Outcomes-Impacts?

Chain of Outcomes SHORTMEDIUMLONG-TERM Seniors increase knowledge of food contamination risks Practice safe cooling of food; food preparation guidelines Lowered incidence of food borne illness Participants increase knowledge and skills in financial management Establish financial goals, use spending plan Reduced debt and increased savings Community increases understanding of childcare needs Residents and employers discuss options and implement a plan Child care needs are met Empty inner city parking lot converted to community garden Youth and adults learn gardening skills, nutrition, food preparation and mgt. Money saved, nutrition improved, residents enjoy greater sense of community

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Focus of Outcomes Individua l –Child, parent, client, resident Group –family, team, community group Agency, organization System Community Child is prepared to enter school; teen uses savings/spending plan Community group has inclusive membership policy; family increases its savings Communications are more open; agency adopted smoke-free policy Family serving agencies share resources Shared community responsibility has increased; youth are valued as contributing members

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop How far out the outcome chain do we go? What is logical? What is realistic? What is meaningful?

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Staff Money Partners Develop parent ed curriculum Deliver 6 interactive sessions Targeted parents attend Parents increase knowledge of child dev Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect INPUTSOUTPUTS OUTCOMES WHICH OUTCOMES??? Research

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Outcome of Interest? Inherently valued outcome (Mohr, 1995) –higher level outcome is immaterial –we are willing to assume that a higher outcome will also be attained if we achieve the outcome of interest Participant valued outcome: if participants experience a change or benefit that makes a a real difference to them (United Way, 1999)

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Assumptions Beliefs about the program –the participants –the way the program will operate –how resources, staff will be engaged –the theory of action

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Assumptions, cont. Faulty assumptions are often the reason for poor results Check and test assumptions –Identify potential barriers for each ‘if-then’ sequence

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Environment–Influential Factors Extension program does not exist in a vacuum Context of the program –politics, family circumstances, cultural milieu, demographics, economics, values, biophysical environment, policies, services What affects the program over which you have little control?

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop What does a logic model look like? Graphic display of boxes and arrows Any shape possible Circular, dynamic Relationships, linkages Level of detail simple complex Multiple models

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Compare examples

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Logic Model: Limitations Represents reality, not reality Programs are not linear Focuses on expected outcomes Challenge of causal attribution Many factors influence outcomes Doesn’t address: Are we doing the right thing?

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Benefits Brings detail to broad goals Shows the ‘chain of events that link inputs to results Builds understanding and consensus Identifies gaps in logic and uncertain assumptions Signals what to evaluate and when Summarizes complex program to communicate with externals

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Building a Logic Model New program Existing program Team; organization Involvement of others Keep it dynamic

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Logic Model: WORKSHEET ASSUMPTIONS INPUTSOUTPUTS Activities Participation OUTCOMES - IMPACT Short Medium Long-term Program Title Situation/Problem

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Check Your Logic Model Are the outcomes really outcomes? Is the longest-term outcome –meaningful? –logical? –realistic? Are the connections between inputs, outputs, and outcomes clear and reasonable? Does it represent research and best practice? Does it represent the program’s purpose; response to the situation?

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop OUTCOMESINPUTSOUTPUTS ActivitiesParticipation ShortMediumLong term Programmatic investments i EVALUATION PLANNING

Where does evaluation fit? Decrease in rates - agency records Actual use - follow-up phone interview Increase in knowledge/skill - post session survey # parents attending/session which parents % of parents Quality of curriculum # of sessions delivered Staff Money Partners Design parent ed curriculum Provide 6 training sessions Targeted parents attend Parents increase knowledge of child dev Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect INPUTSOUTPUTS OUTCOMES EVALUATION: What do you want to know? How will you know it? Providing Leadership for Program Evaluation

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Evaluation Plan Schedule - when/where Method to collect info Source of information Indicators - how will you know it What do you want to know?

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Evaluation Questions – What do you want to know?

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Indicators - How will you know it? The evidence or measures that indicates what you wish to know or see: often multiple indicators are necessary may be quantitative or qualitative culturally appropriate

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Logic model with indicators for Outputs and Outcomes Program implemented Targeted farmers Farmers learn Farmers practice new techniques Farm profitability increases Number of workshops held Number and percent of farmers attending Number and percent who learned content Number and percent who practice the recommendati ons Number and percent show farm profits increase; amount of increase Outputs Outcomes

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop INDICATORS: Examples #, % with savings goal set #, % with debt reduction goal set #, % using spending plan #, % maintaining emergency fund Improved family financial management #, % acres managed according to BMP guidelines Quality of conservation plan implementation Reduction in N and P application rates #, % of Boards with youth participation Increased youth-adult partnerships INDICATORSOUTCOME

Evaluating Your Program Decrease in rates of abuse & neglect among these parents #, % parents using improved skills (specify skills) #, % parents who increased knowledge #, % parents who learned new ways # and %parents who attended each session Quality of curriculum # sessions fully delivered Staff Money Partners Design parent ed curriculum Provide 6 training sessions Targeted parents attend Parents increase knowledge of child dev Parents learn new ways to discipline Parents use improved parenting skills Reduced rates of child abuse & neglect INPUTSOUTPUTS OUTCOMES Indicators

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Source and Method of Data Collection Source of information Participants Parents Teachers Key informants Method of collecting the information Survey Interview Observation End-of-program questionnaire Focus group Records

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Evaluation Plan cont. How will the data be analyzed and interpreted? How will results be shared? To whom, how, when? Who will do what, when, what resources?

University of Wisconsin-Extension Cooperative Extension Program Development & Evaluation Logic Model Workshop Enfin… View evaluation as learning - integrate into the way we work Build evaluation in upfront Ask ‘tough questions’ Make measurement meaningful Be accountable for highest professional standards