Presentation on theme: "The Logic Model in Mental Health Program Development"— Presentation transcript:
1 The Logic Model in Mental Health Program Development Dr. Martin Cortez WesleyLindsey Wilson College
2 A logic model is your program ROAD MAP Where are you going?How will you get there?What will tell you that you’ve arrived?_________________________________________________________A logic model is your program ROAD MAP
3 Provides a common framework Logic model is a…Picture of your program or interventionGraphic representation of the “theory of action” – what is invested, what is done, and what resultsCore of planning and evaluationProvides a common frameworkfor your work
4 MODEL Definition LOGIC the principles of reasoning reasonable the relationship of elements to each other and a wholeMODELsmall object, representing another, often larger object (represents reality, isn’t reality)preliminary pattern serving as a plantentative description of a system or theory that accounts for all of its known propertiesThe American Heritage Dictionary, 2nd Ed
5 The accountability era What gets measured gets doneIf you don’t measure results, you can’t tell success from failureIf you can’t see success, you can’t reward itIf you can’t reward success, you’re probably rewarding failureIf you can’t see success, you can’t learn from itIf you can’t recognize failure, you can’t correct it.If you can demonstrate results, you can win public support.Re-inventing government, Osborne and Gaebler, 1992
6 Language: What do you mean by… Goal = ImpactImpact = Long-term outcomeObjectives (participant focused) = OutcomesActivities = OutputsOutputs may signify “tangible” accomplishments as a result of activities
7 Logic Method Research Provides a common language Helps us differentiate between “what we do” and “results” --- outcomesIncreases understanding about programGuides and helps focus workLeads to improved planning and managementIncreases intentionality and purposeProvides coherence across complex tasks, diverse environments
8 Enhances team workGuides prioritization and allocation of resourcesMotivates staffHelps to identify important variables to measure; use evaluation resources wiselyIncreases resources, opportunities, recognitionSupports replicationOften is required!
9 Logic model is in widespread use Let’s not think that logic modeling is brand new. Actually, the concepts have been around since the late 1960’s in the writings of Suchman, 1967 and Wholey’s evaluability assessment model.It has come to the forefront again, and is being developed and applied in a variety of settings as a result of a variety of factors:Private sector: part of total quality management and performance measurement movementPublic sector, the GPRA has moved all federal agencies to focus on results and link investments to results, not just activities.Non-profit sector is concerned with improving programs to produce valued impacts with the United Way being a frontrunner in outcome measurement using the logic model.International programs. The players in the international arena for a long time have used variations of a logic model. The Log Frame of the US Agency for International Development of the 1980’s is a historical precedent to the current logic modeling discourse.And, professional evaluators have played a prominent role in using and developing the logic model. This is why it is often called an ‘evaluation framework.’ This is a result of evaluators being asked to evaluate impact and finding, too often, that programs didn’t exist, or weren’t being implemented in a way that would achieve the expected impact. Consequently, evaluators began working with programmers to lay out the logic of programs. We see the outgrowth particularly in Chen’s theory-driven evaluation (1990) and Weiss (1997) theory-based evaluation.Private SectorPublic SectorNon-Profit SectorInternational ArenaEvaluators
10 Cook, play, talk, laugh, hike Example: Every day logic model –Family VacationFamily MembersDrive to state parkFamily members learn about each other; family bonds; family has a good timeBudgetSet up campCarCook, play, talk, laugh, hikeCamping EquipmentActually, we use logic models everyday. Let’s look at this…We want to take a family vacation and what we really hope is that we’ll have a good time and enjoy being together. We have had experience and know (our own personal research tells us) that camping is something we all enjoy doing together. So, in order to take a camping trip, we need..If this…, then that….Logic models involve a mental process. A logic model shows the series of connections and logical linkages that is expected to result in achievement of our goal.
11 Example: Professional Counselor Shortage Situation: Eastern Kentucky has a huge shortage of Mental Health Professionals to address the tremendous need in the area.INPUTSOUTPUTSOUTCOMESLindsey Wilson Collegeinvests time andresourcesLWC starts several regional sites to begin to train people to be Mental Health ProfessionalsLocal residents become educated professionals, change service practices, and Mental Health Services improveLet’s apply this to a typical Extension exampleWHAT WE INVESTWHAT WE DOWHAT RESULTS
12 OUTCOMES INPUTS OUTPUTS Situation: Example: One component of a comprehensive parent education and support initiativeSituation:During a county needs assessment, majority of parents reported that they were having difficulty parenting and felt stressed as a resultOUTCOMESINPUTSOUTPUTSParents increase knowledge of child devParents identify appropriate actions to takeDevelop parent ed curriculumStaffImproved child-parent relationsTargetedparentsattendParents better understanding their own parenting styleMoneyDeliver series of interactivesessionsParents use effective parenting practicesStrong familiesPartnersParents gain skills in effective parenting practicesResearchFacilitate support groupsAssumptions:External factors:
13 Example: Smoke free worksites Situation: Secondhand smoke is responsible for lung cancer, respiratory symptoms, cardiovascular disease, and worsens asthma. Public policy change that creates smoke free environments is the best known way to reduce and prevent smoking.InputsOutputsOutcomesAssess worksite tobacco policies and practicesDemonstrations of public support for SF worksitesIncreased awareness of importance of SF worksitesCoalitionTimeDollarsPartnersIncluding youthWorksite owners, managersDevelop community support for SF worksitesSF worksitesIncreased knowledge of SF worksite benefits & optionsSF worksites policies draftedUnionsWorkers; union membersOrganize and implement strategy for targeted worksitesSF worksite policies passedIncreased commitment, support and demand for SF worksitesPublicAdherence to smoke-free policies
14 Logic model of a training workshop Situation: Funder requires grantees to include a logic model in their funding request; grantees have limited understanding of logic models and are unable to fulfill the funding requirementINPUTSOUTPUTSIncrease knowledge of logic modelsIncrease ability to create a useful logic model of programIncrease confidence in using logic modelsOUTCOMESCreate meaningful logic modelsUse logic models in own workFulfill requirement of funder3 hour trainingInteractive activitiesGroup workPracticeQ and ATrainerFundsEquipmentResearch baseTraining curriculumGranteesImproved planningImproved evaluationAccountable here
16 Feedback loops and multi-dimensions Programs aren’t linearFeedback loops and multi-dimensionsINPUTSOUTPUTSOUTCOMESProgram investmentsActivitiesParticipationShortMediumLong-termWhat we investWhat we doWho we reachWhat results
17 Chain of outcomes SHORT MEDIUM LONG-TERM Seniors increase knowledge of foodcontamination risksPractice safe cooling offood; food preparationguidelinesLowered incidence of foodborne illnessParticipants increaseknowledge and skills infinancial managementEstablish financial goals,use spending planReduced debt andincreased savingsCommunity increasesunderstanding ofchildcare needsResidents and employersdiscuss options andimplement a planChild care needs are metEmpty inner city parkinglot converted tocommunity gardenYouth and adults learngardening skills, nutrition,food preparation and mgt.Money saved, nutritionimproved, residents enjoygreater sense ofcommunity
18 Focus of outcomes Individual Group Agency, organization System Child is ready to enter school; farmer implements nutrient management practice Families control spending to maintain family financial stability Agency institutes policy that encourages physical activity of staff Family serving agencies share resources to better meet clientele needs Communities develop and preserve decent safe and affordable housingIndividualChild, parent, client, residentGroupfamily, team, communitygroupAgency, organizationSystemCommunity
19 Writing good outcomesSMART objectives: Specific, measurable, attainable, results-oriented, timedWho/whatChange/desired effectIn whatBy whenFamilies participating in the Family Resource Centerincreasetheir use of community resources and serviceswithin one year of joining4 school boardsadoptpolicies to improve student nutrition and physical activityby Dec 2007
20 What does a logic model look like? Graphic display of boxes and arrows; vertical or horizontalRelationships, linkagesAny shape possibleCircular, dynamicCultural adaptations; storyboardsLevel of detailsimplecomplexMultiple models_________________________________________________________
21 Multiple logic modelsMultiple models may be needed to describe and explain complex initiatives or systems.Multi-level programs: A series of linked models that depict varying levels such as national-state- county levels OR, institution-division-unit levelsMulti-component programs: A series of models to depict various components (goals, sites, target populations) within a comprehensive initiative
22 State level logic model: Reducing and preventing youth tobacco use Outcomes - ImpactInputsActivitiesReachShortMediumLongIncreased awareness of need to eliminate youth access to tobacco products, including tobacco industry tactics, laws, noncompliancePromote community involvement in restricting tobacco access to youthEstablish baseline of current practicesInform/educateEliminate self-serviceFacilitate active enforcement of lawsIncreased compliance and enforcement of laws and policiesDecreased access to tobacco for minorsCommunityParents, CaretakersLaw enforcementRetailerHealth DepartmentCoalition MembersDecreased supply to minorsIncreased commitment to eliminate access/sourcesFundingSocial norms less supportive of youth tobacco useIncreased knowledge and skills in participating in policy changeIncreased # of youth actively engaged in policy changePartnersLocalRegionalStateFacilitate youth involvement in policy changeRecruit youthInvolve youth/adultsEducateCommunity org,BusinessesPolicy makersAdultsYouth serving orgYouthIncreased commitment by youth and adults for youth to participate in policy changeIncreased adoption of policy changes that involve youth in the change processReference: Wisconsin Tobacco Control Program Work Group including Christine Dobbe and Jennifer Leahy of UW-Extension, Based on:Workgroup CDC guidelines and best practicesCIA Sub-Group technical assistance packetCIA Sub-Group reviewersCIA Sub-Group Suggested Goals“Restaurant Ordinances vs. All Worksites Including Restaurants” (CIA Sub-Group, 2003)DPH 2004 Boundary Statement and template objectivesPromote school and community based prevention programs and policiesEstablish baseline of existing resourcesEducateAssist with planning and implementing programs/servicesResearch and best practicesDelayed average age at first use; reduced initiationSchoolsCommunityFamiliesYouth serving orgYouthIncreased knowledge about tobacco dependence; benefits and options for youth prevention (e.g, CDC guidelines, school-family initiatives)Increased # of effective prevention programs or policies adoptedReduced morbidity and mortalityIncreased # of youth participating in prevention programsSee Treating Tobacco Addiction Youth Logic ModelPromote youth cessation services andpoliciesIncreased commitment to adopt effective programs/policies for youth prevention
23 Component Logic Model Youth: Youth Advocating for Policy Change Outcomes - ImpactInputsActivitiesReachShortMediumLongCoalition membersEstablish baseline for policy change in community with help from youthIncreased # youth, community members who:Understand tobacco use issues in their communitiesKnow how to advocate for policy changeIncreased # of youth actively engaged in advocating for policy change in communityCommunity organizations, businesses policy makersIncreased number of tobacco policies in communityTimeEducate youth and adults on policy change options and how to achieve themIncreased # of activities or increased intensity of activities that involve youth to accomplish policy changeAdultsFundingIdentify partners, including youth serving organizations and schools, for engaging youth in policy changePartnersLocalRegionalStateSocial norms less supportive of youth tobacco useYouth serving organizationsIncreased # youth wanting to be involved in advocating for policy changesIncreased adoption of policies that involve youth in the policy changeCounter industry influencePromote clean indoor airDecrease availability of tobacco products in the communityDevelop strategy for and promote engagement of youth in policy changeSchoolsEffective practice strategiesIncreased # youth skilled in being able to advocate for policy changeDelayed average age at first use; reduced initiationAssist with development of youth advocacy skillsYOUTHLocal media outletsPromote community support for youth involvement in community affairs/ policy changeIncreased support for youth involvement in policy change
24 Multi-Org partnership Programs as “systems”Single organizationSHORT-TERM OUTCOMESINPUTSOUTPUTSOrg 1MED-TERM OUTCOMESMulti-Org partnershipOrg 2LONG-TERM OUTCOMESOrg 3CommunityOrg 4Source: Adapted from CDC:
25 Getting started Group process Determine purpose of logic model Who will use it? For what?Involve othersSet boundaries for logic modelUnderstand situationExplore research, knowledge base, what others are doing/have doneGroup process
26 Check your logic model Is it meaningful? Does it make sense? Is it doable?Can it be verified?
27 Limitations Logic Model… Represents reality, is not reality Focuses on expected outcomesChallenge of causal attributionMany factors influence process and outcomesDoesn’t address: Are we doing the right thing?Challenge of causal attribution: Does not suggest/’prove’ that program caused the outcome
28 Where does evaluation fit? From beginning to end
29 What do you want to know? How will you know it? PLANNING: start with the end in mindWhat do you want to know?How will you know it?EVALUATION: check and verify
30 What does evaluation mean to you? Evaluation means asking good, critical questions about programs to improve programs and help them be accountable for the wise use of resources.
31 Logic model and evaluation Needs/asset assessment:What are the characteristics, needs, priorities of target population?What are potential barriers/facilitators?What is most appropriate?Process evaluation:How is program implemented? Fidelity of implementation?Are activities delivered as intended?Are participants being reached as intended?What are participant reactions?Outcome evaluation:To what extent are desired changes occurring? For whom?Is the program making a difference?What seems to work? Not work?What are unintended outcomes?
32 EVALUATION: What do you (and others) want to know about this program? Parents increase knowledge of child devDevelop parent ed curriculumParents identify appropriate actions to takeStaffTargetedparentsattendImproved child-parent relationsMoneyDeliver series of interactivesessionsParents better understand their own parenting stylePartnersParents use effective parenting practicesResearchParents gain skills in effective parenting practicesStrong familiesFacilitate support groupsWhat amount of $ and time were invested?How many sessions were actually delivered? How effectively?# and quality of support groups?Who/how many attended/did not attend? Did they attend all sessions?Supports groups? Were they satisfied – will they come again?To what extent did knowledge and skills increase? For whom? Why? What else happened?To what extent did behaviorschange? For whom? Why? What else happened?To what extent are relations improved? Does this result in stronger families?
33 Prioritize Lots of questions and so little time Prioritize evaluation questionsEvaluation purposeNeedContextProcessOutcomesStakeholder needs
34 You – staff Participants Funder Who wants to know what about your program?WHO might use the evaluation?WHAT do they want to know?HOW will they use the info?You – staffParticipantsFunder
35 Developing an evaluation plan based on your logic model 1. Focus:2. Questions3. Indicators4. Timing5. Data collectionSourcesMethodsSampleInstrumentsInputsOutputsOutcomes
36 Logic model and reporting University of Wisconsin-Extension, Program Development and Evaluation
37 “We build the road and the road builds us.” -Sri Lankan saying