1 Frameworks for program planning and evaluation: Problem Analysis and Logic Models Geraldine Oliva M.D. , MPHRoberto Ariel Vargas, MPHJames Rouse-Iñiguez, MAPaula Fleisher, MAJune 2013
2 Today’s Agenda Why Evaluation?! Using a socio-ecological problem analysis framework to identify “causal pathways”Developing strategies to address a problem based on upstream causes and risksDeveloping a logic model for program planning and evaluation
3 Why Evaluation?! Root word: value Whose values? Funders, clients, organization, etc.It helps us tell our storyIt helps us understand what we are doing welland what we can improve…
4 “Traditional” Program Planning Cycle ConvenePublic Health/ Community CoalitionAssess Community /MCAHResources & Strengths/CapacityAnalyze Problem & Select InterventionsEvaluate/ MeasurePerformancePlan & Implement ProgramsDevelopObjectivesAssess & PrioritizeHealth Status/ProblemsThe information of these new paradigm impacts all the steps of the planning process. At step one in recognition of the importance of social determinants of health it makes sense to include a broader array of stakeholders that can assist in providing information on economic, social and policy issues that affect the MCAH population. During the assessment process data analysts will need to gather data at the individual, and community level on SES factors. But the main impact to the planning cycle will be during the analysis phase when the group is looking in depth at priority issues to develop strategies for change. It will also be critical at the intervention level to look at actions that can impact the higher level health determinants and also to consider act ivies that span the life course.
5 New Paradigms for Understanding Persistent Disparities Emerging evidence: traditional clinical and public health efforts targeting individuals have not impacted health disparitiesMulti-level analysis has demonstrated the significance of social determinants of health in explaining many of these disparitiesRe-emergence of “life course model”, importance of prevention and intervention along age spectrum
8 Lifecourse Model as a Cycle Mother overweight/diabeticPoverty/fast food/food habits in young womanInfant born LGA/Abnormal GTAdolescent with increased # fat cells/abnormal GTWorking Mom-Infant FedFormulaSchool with no PE, unsafe parkslatch key child watches TV /gets more overweightFamily Culture - high fat, high sugar diet- child overweightUCSF Family Health Outcomes Project
9 How do we Incorporate these New Paradigms into a Community/Public Health Planning Framework ?
10 African-American, Latino Children Drink More Sugary Fruit Juice Than Their White Peers Share this story:Share PrintBy Juliana Bunim on May 30, 2013While there has been a steep decline in kids’ consumption of sugar-sweetened beverages in California, African-American and Latino children may be replacing soda with 100 percent fruit juice while their white peers are not, according to a new study from UC San Francisco.The study was the first to compare trends of sugar-sweetened beverages and 100 percent juice consumption in California. “The decrease in the consumption of sugar-sweetened beverages among kids is a promising public health trend,” said Amy Beck, MD, MPH, lead author and pediatrician at UCSF Benioff Children’s Hospital and San Francisco General Hospital and Trauma Center. “But it is concerning that minority children are increasing their consumption of 100 percent fruit juice, which often has just as much sugar as soda.”…Fruit juice is available in schools through the National School Lunch Program, which provided low-cost or free lunches to more than 31 million children nationally each school day in Juice also is distributed to parents through the Women, Infants and Children (WIC) program, a federal grant program for low-income, nutritionally at-risk women and children up to age 5 and through the Child Care Food Program.
11 Seeing solutions with “Ecological Lenses” At the individual level: Nutrition information (for parent or child)At the family level: Nutrition education for parent and childAt the community level: a media campaign targeting familiesAt the organizational level: commit to soda AND juice free zones in community centers, provide access to fruit and clean waterAt the policy level: change WIC policy to provide whole fruit and water filters, eliminate Juice vouchers for WICOREliminate fruit juice from subsidized school lunches; increase fruit and provide filling stations and water bottles in schools
12 Rationale for Doing a Formal Problem Analysis? In order to identify effective intervention strategies it is necessary to understand the complex array of underlying factors that can impact a health outcome and how they relate to one anotherUsing a multilevel socio-ecological framework ensures that upstream factors are includedRelating upstream precursors to downstream outcomes forces us to explore the pathways by which upstream factors operate in a specific situation
13 Steps in a Problem Analysis 1. Examine epidemiologic data2. Examine literature and consult experts (community and science)3. Determine extent to which these factors are active in the community
14 Steps in a Problem Analysis 4. Determine relative contribution of each identified factor5. Identify the interrelationships among factors – causal pathways6. Determine the most effective points in the causal pathways for intervention
15 Psychological Factors A Generic Framework for Health Problem AnalysisSocial/Economic/Policy LevelSESSafetyEducationCultureEnvironmentEconomyHealth Care PoliciesFamily/Community/Institutions LevelSchool/WorkplaceFamily/HouseholdCommunityHealth Care/ProvidersIndividual LevelHealth Status/Medical ConditionsGenetic/BiologicalPsychological FactorsCognitive FactorsHealth BehaviorsIdentified Problem
17 Too much screen time (TV computer) Insufficiency of physical activity SES/PolicyMarketing by fast food industryUnsafe neighborhood/streetsInadequate subsidized healthInsurance for the poorPovertyCity planning Policies that limit green space in some cities and neighborhoodsLack of funding for educationFamily/communityNo opportunities for exercise in school or recreation areasFamily lack of health InsuranceHealth serviceslack of knowledge of nutritionHigh costs for treatmentFamilies limited incomeParental knowledge and behavior ( time with kids, food prep, own eating and exercise habits)Unavailability of healthy food choices in local stores , schools and restaurantsToo much screen time (TV computer)IndividualUnhealthy dietInsufficiency of physical activityPROBLEM:Childhood Obesity
18 Linkages precursors and problem Definition: The association between Review the literatureConsult expertsCommunityScienceConsult stakeholdersAnalyze your data
19 Too much screen time (TV computer) Insufficient physical activity SES/PolicyMarketing by fast food industryUnsafe neighborhood/streetsInadequate subsidized healthInsurance for the poorPovertyCity planning Policies that limit green space in some cities and neighborhoodsLack of funding for educationFamily/communityNo opportunities for exercise in school or recreation areasFamily lack of health InsuranceHealth serviceslack of knowledge of nutritionHigh costs for treatmentFamilies limited incomeParental knowledge and behavior ( time with kids, food prep, own eating and exercise habits)Unavailability of healthy food choices in local stores , schools and restaurantsToo much screen time (TV computer)IndividualUnhealthy dietInsufficient physical activityPROBLEM:Childhood Obesity
20 Diagramming Causal Pathways to Identify Strategic Interventions Unsafe neighborhood streetsInterventionOverweightNo opportunities for exercise in school or recreation areasSay this is the pathway for low IZ rates. A survey of parents show they don’t perceive communicable childhood diseases to be a threat.Where’s the intervention point?What if you found out that two providers have poor tracking methods and don’t make reminder calls to parents -- what’s an intervention in that situation?Tell Gerry’s story re: LBW in SF. Intensive programs of enhanced PNC. Didn’t make a dent in the outcomes at one clinic. Turned out, 65% of the pregnant women there smoked. Revised intervention to include smoking cessation.Inadequate physical activityToo much screen time (TV computer)
22 Gang Membership/Incarceration Problem Analysis – Homey ClientsSocial Determinants crime/gun policies/ prevention $Economic policy Poverty / no jobs or bias in hiringRacism/Destruction ofindigenous culture $ for schools/low standards/legacy of colonialismFamily/Community/Local InstitutionsPolice racial profilingCommunity-lack of social capital/ cohesion/pride/trustFamily disintegrationParents don’t care/use drug/don’t supervise/not educated/poor/lack cultural pride Jobs/lack of training opportunitiesBarriers to accessing services:Language/Cultural/ $Transportation/LegalSchools push kids out/racism in staff/poor quality/lack of cultural affinityIndividualsAlcohol/drug useUnsafe sex/ bad food/no exerciseMental illness/PTSD/ self esteem/ resiliency/ trust/ sense of belongingFAS/ADD/ADHD/other drug exposureLearning difficulties/brain trauma/school failure/no college prepPROBLEM:Gang Membership/Incarceration
23 HOMEY- Kapuli Causal Pathway: Program Intervention Points Communities:Racism in schoolsRacism by policeLack of cultural identity in family/communityfamily disintegrationKids:Mental health problemsSchool failureLow self esteem lack resiliencyNo sense of true cultural identityNo sense of trust or belongingGangs/JailSociety:RacismLack of respectfor indigenous culturesSchool failureSelf- Destructive BehaviorsKapuli Interventions
24 ENVIRONMENTAL FACTORS MOMEY-Kapuli Logic ModelLearning about history of social justice movements will give them tools for changeLearning about impact of racism/colonialism will give kids an undestanding of their own historyEpisodes of tragedy due to racism in their communityMedia/movie about examples of racism/harassment by policeENVIRONMENTAL FACTORSStreet outreachEducate about history of social justice movements for people of color/indigenous10 strategies for changeTeach about racism/colonialismPositive response during recap activityActive participation in groupSuccessful school/CBO outreachSchoolsCBO’sKids from streetsCome backJoin political campaignsAttend ceremoniesExpress value of social actionIncrease self esteem/resiliency/ capacity for actionKids return to schoolKids get jobsKids stay out of gangsKids stay out of jailStaffCameronSpaceCore MembersFundsINPUTSOUTPUTSOUTCOMES – IMPACTActivitiesParticipationShortMediumLong-TermASSUMPTIONS:Adapted from
25 Another Example: Obesity prevention/ intervention in the Bayview Hunters Point of SF Designed in 2011, in partnership between:UCSFSan Francisco Department of Public healthBVHP residentsother stakeholdersSee handouts for problem analysis and logic modelsAs of 2013, focus is on:Reducing consumption of sugar sweetened beveragesActivating community park space
26 BenefitsClearly documents the decision making process for the group and for othersCan use simplified problem analysis or causal pathway diagram to communicate rationale for intervention strategy to policy makers and the public
27 Logic Model - DefinedLogic models are a common tool used by evaluators and program plannersLogic models are graphic depictions of the relationship between a program’s activities and its intended outcomes.U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.
28 Logic Model – Terms Relationship Intended Logic models describe activities that comprise the program, the inter-relationship of those activities, and the link between these and outcomes.IntendedLogic models depict “intended” outcomes of a program’s activities, rather than reality at any point in time. As the starting point for evaluation and planning, the model serves as an “outcomes roadmap” that shows the logic behind the program (describing why it will work). Of all the activities that might address this issue, these were chosen because they will work, and we have (or know) the resources needed . Over time, evaluation, research, and experience will help us learn what works, and the model will evolve.U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.
29 Logic Model - Components InputsResources that go into the program and are required to make it happenActivitiesevents or actions done by the program and its staffOutputsDirect products of program activities, often measured in countable terms (e.g., #of zumba classes; # of participants).OutcomesThe changes that result from the program’s activities and outputs, often expressed as short-term, intermediate, and long-term outcomes.
30 Basic Program Logic Model U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.
31 Let’s build a logic model! Using the problem you’ve identified, begin by identifying the change you want to make (outcome of interest)What is your “theory of change”? Or, what do you think it would take to make change? This will be the underlying logic that guides what activities your program doesWhat resources do you need (inputs)?
32 Q and A What was clarified for you; any “aha!” moments? What still concerns or challenges you?
33 Contact InformationCommunity Engagement and Health Policy Program, CTSI/UCSF