Frameworks for program planning and evaluation: Problem Analysis and Logic Models Geraldine Oliva M.D. , MPH Roberto Ariel Vargas, MPH James Rouse-Iñiguez, MA Paula Fleisher, MA June 2013

Today’s Agenda Why Evaluation?! Using a socio-ecological problem analysis framework to identify “causal pathways” Developing strategies to address a problem based on upstream causes and risks Developing a logic model for program planning and evaluation

Why Evaluation?! Root word: value Whose values? Funders, clients, organization, etc. It helps us tell our story It helps us understand what we are doing well and what we can improve…

“Traditional” Program Planning Cycle Convene Public Health/ Community Coalition Assess Community /MCAH Resources & Strengths/Capacity Analyze Problem & Select Interventions Evaluate/ Measure Performance Plan & Implement Programs Develop Objectives Assess & Prioritize Health Status/Problems The information of these new paradigm impacts all the steps of the planning process. At step one in recognition of the importance of social determinants of health it makes sense to include a broader array of stakeholders that can assist in providing information on economic, social and policy issues that affect the MCAH population. During the assessment process data analysts will need to gather data at the individual, and community level on SES factors. But the main impact to the planning cycle will be during the analysis phase when the group is looking in depth at priority issues to develop strategies for change. It will also be critical at the intervention level to look at actions that can impact the higher level health determinants and also to consider act ivies that span the life course.

New Paradigms for Understanding Persistent Disparities Emerging evidence: traditional clinical and public health efforts targeting individuals have not impacted health disparities Multi-level analysis has demonstrated the significance of social determinants of health in explaining many of these disparities Re-emergence of “life course model”, importance of prevention and intervention along age spectrum

Socio-Ecological Models

Lifecourse Model

Lifecourse Model as a Cycle Mother overweight/diabetic Poverty/fast food/food habits in young woman Infant born LGA/Abnormal GT Adolescent with increased # fat cells/ abnormal GT Working Mom-Infant Fed Formula School with no PE, unsafe parks latch key child watches TV / gets more overweight Family Culture - high fat, high sugar diet- child overweight UCSF Family Health Outcomes Project

How do we Incorporate these New Paradigms into a Community/Public Health Planning Framework ?

African-American, Latino Children Drink More Sugary Fruit Juice Than Their White Peers Share this story: Share  Print By Juliana Bunim on May 30, 2013 Email While there has been a steep decline in kids’ consumption of sugar-sweetened beverages in California, African-American and Latino children may be replacing soda with 100 percent fruit juice while their white peers are not, according to a new study from UC San Francisco. The study was the first to compare trends of sugar-sweetened beverages and 100 percent juice consumption in California.   “The decrease in the consumption of sugar-sweetened beverages among kids is a promising public health trend,” said Amy Beck, MD, MPH, lead author and pediatrician at UCSF Benioff Children’s Hospital and San Francisco General Hospital and Trauma Center. “But it is concerning that minority children are increasing their consumption of 100 percent fruit juice, which often has just as much sugar as soda.” …Fruit juice is available in schools through the National School Lunch Program, which provided low-cost or free lunches to more than 31 million children nationally each school day in 2011. Juice also is distributed to parents through the Women, Infants and Children (WIC) program, a federal grant program for low-income, nutritionally at-risk women and children up to age 5 and through the Child Care Food Program.

Seeing solutions with “Ecological Lenses” At the individual level: Nutrition information (for parent or child) At the family level: Nutrition education for parent and child At the community level: a media campaign targeting families At the organizational level: commit to soda AND juice free zones in community centers, provide access to fruit and clean water At the policy level: change WIC policy to provide whole fruit and water filters, eliminate Juice vouchers for WIC OR Eliminate fruit juice from subsidized school lunches; increase fruit and provide filling stations and water bottles in schools

Rationale for Doing a Formal Problem Analysis? In order to identify effective intervention strategies it is necessary to understand the complex array of underlying factors that can impact a health outcome and how they relate to one another Using a multilevel socio-ecological framework ensures that upstream factors are included Relating upstream precursors to downstream outcomes forces us to explore the pathways by which upstream factors operate in a specific situation

Steps in a Problem Analysis 1. Examine epidemiologic data 2. Examine literature and consult experts (community and science) 3. Determine extent to which these factors are active in the community

Steps in a Problem Analysis 4. Determine relative contribution of each identified factor 5. Identify the interrelationships among factors – causal pathways 6. Determine the most effective points in the causal pathways for intervention

Psychological Factors A Generic Framework for Health Problem Analysis Social/Economic/Policy Level SES Safety Education Culture Environment Economy Health Care Policies Family/Community/Institutions Level School/Workplace Family/Household Community Health Care/Providers Individual Level Health Status/ Medical Conditions Genetic/Biological Psychological Factors Cognitive Factors Health Behaviors Identified Problem

Too much screen time (TV computer) Insufficiency of physical activity SES/Policy Marketing by fast food industry Unsafe neighborhood/streets Inadequate subsidized health Insurance for the poor Poverty City planning Policies that limit green space in some cities and neighborhoods Lack of funding for education Family/community No opportunities for exercise in school or recreation areas Family lack of health Insurance Health services lack of knowledge of nutrition High costs for treatment Families limited income Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits) Unavailability of healthy food choices in local stores , schools and restaurants Too much screen time (TV computer) Individual Unhealthy diet Insufficiency of physical activity PROBLEM: Childhood Obesity

Linkages precursors and problem Definition: The association between Review the literature Consult experts Community Science Consult stakeholders Analyze your data

Too much screen time (TV computer) Insufficient physical activity SES/Policy Marketing by fast food industry Unsafe neighborhood/streets Inadequate subsidized health Insurance for the poor Poverty City planning Policies that limit green space in some cities and neighborhoods Lack of funding for education Family/community No opportunities for exercise in school or recreation areas Family lack of health Insurance Health services lack of knowledge of nutrition High costs for treatment Families limited income Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits) Unavailability of healthy food choices in local stores , schools and restaurants Too much screen time (TV computer) Individual Unhealthy diet Insufficient physical activity PROBLEM: Childhood Obesity

Diagramming Causal Pathways to Identify Strategic Interventions Unsafe neighborhood streets Intervention Overweight No opportunities for exercise in school or recreation areas Say this is the pathway for low IZ rates. A survey of parents show they don’t perceive communicable childhood diseases to be a threat. Where’s the intervention point? What if you found out that two providers have poor tracking methods and don’t make reminder calls to parents -- what’s an intervention in that situation? Tell Gerry’s story re: LBW in SF. Intensive programs of enhanced PNC. Didn’t make a dent in the outcomes at one clinic. Turned out, 65% of the pregnant women there smoked. Revised intervention to include smoking cessation. Inadequate physical activity Too much screen time (TV computer)

Gang Membership/Incarceration Problem Analysis – Homey Clients Social Determinants  crime/gun policies/  prevention $ Economic policy Poverty / no jobs or bias in hiring Racism/Destruction of indigenous culture  $ for schools/low standards/legacy of colonialism Family/Community/Local Institutions Police racial profiling Community-lack of social capital/ cohesion/pride/trust Family disintegration Parents don’t care/use drug/don’t supervise/not educated/poor/lack cultural pride  Jobs/lack of training opportunities Barriers to accessing services: Language/Cultural/ $ Transportation/Legal Schools push kids out/racism in staff/poor quality/lack of cultural affinity Individuals Alcohol/drug use Unsafe sex/ bad food/no exercise Mental illness/PTSD/  self esteem/  resiliency/  trust/ sense of belonging FAS/ADD/ADHD/other drug exposure Learning difficulties/brain trauma/school failure/no college prep PROBLEM: Gang Membership/Incarceration

HOMEY- Kapuli Causal Pathway: Program Intervention Points Communities: Racism in schools Racism by police Lack of cultural identity in family/community family disintegration Kids: Mental health problems School failure Low self esteem lack resiliency No sense of true cultural identity No sense of trust or belonging Gangs/Jail Society: Racism Lack of respect for indigenous cultures School failure Self- Destructive Behaviors Kapuli Interventions

ENVIRONMENTAL FACTORS MOMEY-Kapuli Logic Model Learning about history of social justice movements will give them tools for change Learning about impact of racism/colonialism will give kids an undestanding of their own history Episodes of tragedy due to racism in their community Media/movie about examples of racism/harassment by police ENVIRONMENTAL FACTORS Street outreach Educate about history of social justice movements for people of color/indigenous 10 strategies for change Teach about racism/colonialism Positive response during recap activity Active participation in group Successful school/CBO outreach Schools CBO’s Kids from streets Come back Join political campaigns Attend ceremonies Express value of social action Increase self esteem/resiliency/ capacity for action Kids return to school Kids get jobs Kids stay out of gangs Kids stay out of jail Staff Cameron Space Core Members Funds INPUTS OUTPUTS OUTCOMES – IMPACT Activities Participation Short Medium Long-Term ASSUMPTIONS: Adapted from

Another Example: Obesity prevention/ intervention in the Bayview Hunters Point of SF Designed in 2011, in partnership between: UCSF San Francisco Department of Public health BVHP residents other stakeholders See handouts for problem analysis and logic models As of 2013, focus is on: Reducing consumption of sugar sweetened beverages Activating community park space

Benefits Clearly documents the decision making process for the group and for others Can use simplified problem analysis or causal pathway diagram to communicate rationale for intervention strategy to policy makers and the public

Logic Model - Defined Logic models are a common tool used by evaluators and program planners Logic models are graphic depictions of the relationship between a program’s activities and its intended outcomes. U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

Logic Model – Terms Relationship Intended Logic models describe activities that comprise the program, the inter-relationship of those activities, and the link between these and outcomes. Intended Logic models depict “intended” outcomes of a program’s activities, rather than reality at any point in time. As the starting point for evaluation and planning, the model serves as an “outcomes roadmap” that shows the logic behind the program (describing why it will work). Of all the activities that might address this issue, these were chosen because they will work, and we have (or know) the resources needed . Over time, evaluation, research, and experience will help us learn what works, and the model will evolve. U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

Logic Model - Components Inputs Resources that go into the program and are required to make it happen Activities events or actions done by the program and its staff Outputs Direct products of program activities, often measured in countable terms (e.g., #of zumba classes; # of participants). Outcomes The changes that result from the program’s activities and outputs, often expressed as short-term, intermediate, and long-term outcomes.

Basic Program Logic Model U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

Let’s build a logic model! Using the problem you’ve identified, begin by identifying the change you want to make (outcome of interest) What is your “theory of change”? Or, what do you think it would take to make change? This will be the underlying logic that guides what activities your program does What resources do you need (inputs)?

Q and A What was clarified for you; any “aha!” moments? What still concerns or challenges you?

Contact Information Community Engagement and Health Policy Program, CTSI/UCSF 415-206-4048 rousej@fcm.ucsf.edu paula.fleisher@ucsf.edu roberto.vargas@ucsf.edu www.sfhip.org