Presentation on theme: "Logic models to enhance program performance"— Presentation transcript:
1Logic models to enhance program performance Ellen Taylor-Powell, Ph.D. Evaluation Specialist University of Wisconsin- Extension-Cooperative Extension
2A logic model is your program ROAD MAP Where are you going?How will you get there?What will tell you that you’ve arrived?A logic model is your program ROAD MAP_________________________________________________________
3Logic model is a…Picture of your program or interventionGraphic representation of the “theory of action” – what is invested, what is done, and what resultsCore of planning and evaluationProvides a common framework for your work
4LOGIC MODEL Definition the principles of reasoning reasonable the relationship of elements to each other and a wholeMODELsmall object, representing another, often larger object (represents reality, isn’t reality)preliminary pattern serving as a plantentative description of a system or theory that accounts for all of its known propertiesThe American Heritage Dictionary, 2nd Ed
5The accountability era What gets measured gets doneIf you don’t measure results, you can’t tell success from failureIf you can’t see success, you can’t reward itIf you can’t reward success, you’re probably rewarding failureIf you can’t see success, you can’t learn from itIf you can’t recognize failure, you can’t correct it.If you can demonstrate results, you can win public support.Re-inventing government, Osborne and Gaebler, 1992
6Logic model is in widespread use Private SectorPublic Sector: GPRANon-Profit SectorInternational ArenaEvaluatorsLet’s not think that logic modeling is brand new. Actually, the concepts have been around since the late 1960’s in the writings of Suchman, 1967 and Wholey’s evaluability assessment model.It has come to the forefront again, and is being developed and applied in a variety of settings as a result of a variety of factors:Private sector: part of total quality management and performance measurement movementPublic sector, the GPRA has moved all federal agencies to focus on results and link investments to results, not just activities.Non-profit sector is concerned with improving programs to produce valued impacts with the United Way being a frontrunner in outcome measurement using the logic model.International programs. The players in the international arena for a long time have used variations of a logic model. The Log Frame of the US Agency for International Development of the 1980’s is a historical precedent to the current logic modeling discourse.And, professional evaluators have played a prominent role in using and developing the logic model. This is why it is often called an ‘evaluation framework.’ This is a result of evaluators being asked to evaluate impact and finding, too often, that programs didn’t exist, or weren’t being implemented in a way that would achieve the expected impact. Consequently, evaluators began working with programmers to lay out the logic of programs. We see the outgrowth particularly in Chen’s theory-driven evaluation (1990) and Weiss (1997) theory-based evaluation.
7Cook, play, talk, laugh, hike Example: Every day logic model –Family VacationFamily MembersDrive to state parkFamily members learn about each other; family bonds; family has a good timeBudgetSet up campCarCook, play, talk, laugh, hikeCamping EquipmentActually, we use logic models everyday. Let’s look at this…We want to take a family vacation and what we really hope is that we’ll have a good time and enjoy being together. We have had experience and know (our own personal research tells us) that camping is something we all enjoy doing together. So, in order to take a camping trip, we need..If this…, then that….Logic models involve a mental process. A logic model shows the series of connections and logical linkages that is expected to result in achievement of our goal.
8Example: Financial management program Situation: Individuals with limited knowledge and skills in basic financial management are unable to meet their financial goals and manage money to meet their needs.INPUTSOUTPUTSOUTCOMESExtension invests time and resourcesWe conduct a variety of educational activitiestargeted to individuals who participateParticipants gain knowledge, change practices and have improved financial well-beingLet’s apply this to a typical Extension exampleWHAT WE INVESTWHAT WE DOWHAT RESULTS
9Example: One component of a comprehensive parent education and support initiative Situation:During a county needs assessment, majority of parents reported that they were having difficulty parenting and felt stressed as a resultOUTCOMESINPUTSOUTPUTSParents increase knowledge of child devParents identify appropriate actions to takeDevelop parent ed curriculumStaffImproved child-parent relationsTargetedparentsattendMoneyParents better understanding their own parenting styleDeliver series of interactivesessionsParents use effective parenting practicesStrong familiesPartnersParents gain skills in effective parenting practicesResearchFacilitate support groupsAssumptions:External factors:
10Example: Smoke free worksites Situation: Secondhand smoke is responsible for lung cancer, respiratory symptoms, cardiovascular disease, and worsens asthma. Public policy change that creates smoke free environments is the best known way to reduce and prevent smoking.InputsOutputsOutcomesAssess worksite tobacco policies and practicesIncreased awareness of importance of SF worksitesDemonstrations of public support for SF worksitesCoalitionTimeDollarsPartnersIncluding youthWorksite owners, managersDevelop community support for SF worksitesIncreased knowledge of SF worksite benefits & optionsSF worksitesSF worksites policies draftedUnionsWorkers; union membersOrganize and implement strategy for targeted worksitesSF worksite policies passedIncreased commitment, support and demand for SF worksitesPublicAdherence to smoke-free policies
11Example: Logic model training workshop Situation: Funder requires grantees to include a logic model in funding request; grantees have limited understanding of logic models and are unable to fulfill the funding requirementINPUTSOUTPUTSOUTCOMESIncrease knowledge of logic modelsIncrease ability to create a meaningful logic model of programIncrease confidence in using logic modelsUse logic models in planning and evaluation – in your own workModel quality logic model practiceTrainerBudgetEquipmentResearch baseTraining curriculum3 hour trainingInteractive activitiesGroup workPracticeQ and AImproved planning – programs achieve positive resultsImproved evaluation - more credible and useful dataGranteesAccountable here
13Connecting outputs to outcomes is a challenge “I think you should be more explicit here in Step Two.”
14Feedback loops and multi-dimensions Programs aren’t linearFeedback loops and multi-dimensionsINPUTSOUTPUTSOUTCOMESProgram investmentsActivitiesParticipationShortMediumLong-termWhat we investWhat we doWho we reachWhat results
15Chain of outcomes SHORT MEDIUM LONG-TERM Seniors increase knowledge of foodcontamination risksPractice safe cooling offood; food preparationguidelinesLowered incidence of foodborne illnessParticipants increaseknowledge and skills infinancial managementEstablish financial goals,use spending planReduced debt andincreased savingsCommunity increasesunderstanding ofchildcare needsResidents and employersdiscuss options andimplement a planChild care needs are metEmpty inner city parkinglot converted tocommunity gardenYouth and adults learngardening skills, nutrition,food preparation and mgt.Money saved, nutritionimproved, residents enjoygreater sense ofcommunity
16Focus of outcomes Individual Group Agency, organization System Child is ready to enter school; farmer implements nutrient management practiceFamilies control spending to maintain family financial stabilityAgency institutes policy that encourages physical activity of staffFamily serving agencies share resources to better meet clientele needsCommunities develop and preserve decent safe and affordable housingIndividualChild, parent, client, residentGroupfamily, team, communitygroupAgency, organizationSystemCommunity
17Writing good outcomesSMART objectives: Specific, measurable, attainable, results-oriented, timedWho/whatChange/desired effectIn whatBy whenFamilies participating in the Family Resource Centerincreasetheir use of community resources and serviceswithin one year of joining4 school boardsadoptpolicies to improve student nutrition and physical activityby Dec 2005
18What does a logic model look like? Graphic display of boxes and arrows; vertical or horizontalRelationships, linkagesAny shape possibleCircular, dynamicCultural adaptations; storyboardsLevel of detailSimpleComplexMultiple models_________________________________________________________
19Multiple logic modelsMultiple models may be needed to describe and explain complex initiatives or systems.Multi-level programs: A series of linked models that depict varying levels such as national-state-county levels OR, institution-division-unit levelsMulti-component programs: A series of models to depict various components (goals, sites, target populations) within a comprehensive initiative
20State level logic model: Reducing and preventing youth tobacco use Outcomes - ImpactInputsActivitiesReachShortMediumLongIncreased awareness of need to eliminate youth access to tobacco products, including tobacco industry tactics, laws, noncompliancePromote community involvement in restricting tobacco access to youthEstablish baseline of current practicesInform/educateEliminate self-serviceFacilitate active enforcement of lawsIncreased compliance and enforcement of laws and policiesDecreased access to tobacco for minorsCommunityParents, CaretakersLaw enforcementRetailerHealth DepartmentCoalition MembersDecreased supply to minorsIncreased commitment to eliminate access/sourcesFundingSocial norms less supportive of youth tobacco useIncreased knowledge and skills in participating in policy changeIncreased # of youth actively engaged in policy changePartnersLocalRegionalStateFacilitate youth involvement in policy changeRecruit youthInvolve youth/adultsEducateCommunity org,BusinessesPolicy makersAdultsYouth serving orgYouthIncreased commitment by youth and adults for youth to participate in policy changeIncreased adoption of policy changes that involve youth in the change processPromote school and community based prevention programs and policiesEstablish baseline of existing resourcesEducateAssist with planning and implementing programs/servicesResearch and best practicesDelayed average age at first use; reduced initiationReference: Wisconsin Tobacco Control Program Work Group including Christine Dobbe and Jennifer Leahy of UW-Extension, Based on:Workgroup CDC guidelines and best practicesCIA Sub-Group technical assistance packetCIA Sub-Group reviewersCIA Sub-Group Suggested Goals“Restaurant Ordinances vs. All Worksites Including Restaurants” (CIA Sub-Group, 2003)DPH 2004 Boundary Statement and template objectivesSchoolsCommunityFamiliesYouth serving orgYouthIncreased knowledge about tobacco dependence; benefits and options for youth prevention (e.g, CDC guidelines, school-family initiatives)Increased # of effective prevention programs or policies adoptedIncreased # of youth participating in prevention programsReduced morbidity and mortalitySee Treating Tobacco Addiction Youth Logic ModelPromote youth cessation services andpoliciesIncreased commitment to adopt effective programs/policies for youth prevention
21Component Logic Model Youth: Youth Advocating for Policy Change Outcomes - ImpactInputsActivitiesReachShortMediumLongCoalition membersEstablish baseline for policy change in community with help from youthIncreased # youth, community members who:Understand tobacco use issues in their communitiesKnow how to advocate for policy changeIncreased # of youth actively engaged in advocating for policy change in communityCommunity organizations, businesses policy makersIncreased number of tobacco policies in communityTimeEducate youth and adults on policy change options and how to achieve themIncreased # of activities or increased intensity of activities that involve youth to accomplish policy changeAdultsFundingIdentify partners, including youth serving organizations and schools, for engaging youth in policy changePartnersLocalRegionalStateSocial norms less supportive of youth tobacco useYouth serving organizationsIncreased # youth wanting to be involved in advocating for policy changesIncreased adoption of policies that involve youth in the policy changeCounter industry influencePromote clean indoor airDecrease availability of tobacco products in the communityDevelop strategy for and promote engagement of youth in policy changeEffective practice strategiesSchoolsIncreased # youth skilled in being able to advocate for policy changeDelayed average age at first use; reduced initiationAssist with development of youth advocacy skillsYOUTHLocal media outletsPromote community support for youth involvement in community affairs/ policy changeIncreased support for youth involvement in policy change
22Getting started Determine purpose of logic model Involve others Who will use it? For what?Involve othersSet boundaries for logic modelUnderstand situationExplore research, knowledge base, what others are doing/have doneGroup process
23Check your logic model Is it meaningful? Does it make sense? Is it doable?Can it be verified?
24Limitations Logic Model… Represents reality, is not reality Focuses on expected outcomesChallenge of causal attributionMany factors influence process and outcomesDoesn’t address: Are we doing the right thing?Challenge of causal attribution: Does not suggest/’prove’ that program caused the outcome
25Where does evaluation fit? From beginning to end
26What do you want to know? How will you know it? PLANNING: start with the end in mindWhat do you want to know?How will you know it?EVALUATION: check and verify
27What does evaluation mean to you? Evaluation means asking good, critical questions about programs to improve programs and help them be accountable for the wise use of resources.
28EVALUATION: What do you (and others) want to know about this program? StaffMoneyPartnersDevelop parent ed curriculumDeliver series of interactivesessionsParents increase knowledge of child devParents better understand their own parenting styleParents use effective parenting practicesImproved child-parent relationsResearchFacilitate support groupsParents gain skills in effective parenting practicesParents identify appropriate actions to takeStrong familiesTargetedparentsattendEVALUATION: What do you (and others) want to know about this program?To what extent are relations improved? Does this result in stronger families?To what extent did behaviorschange? For whom? Why? What else happened?To what extent did knowledge and skills increase? For whom? Why? What else happened?Did all parents attend that we intended? Who did/not not?Did they attend all sessions?Were all sessions delivered? How effectively?What amount of $ and time were invested?
29Prioritize Lots of questions and so little time Prioritize evaluation questionsEvaluation purposeNeedContextProcessOutcomesStakeholder needs
30Who wants to know what about your program? WHO might use the evaluation?WHAT do they want to know?HOW will they use the info?You – staffParticipantsFunder
31Developing an evaluation plan based on your logic model 1. Focus:2. Questions3. Indicators4. Timing5. Data collectionSourcesMethodsSampleInstrumentsInputsOutputsOutcomes