1. Frameworks for program planning and evaluation: Problem Analysis and Logic Models
Geraldine Oliva M.D. , MPH
Roberto Ariel Vargas, MPH
James Rouse-Iñiguez, MA
Paula Fleisher, MA
June 2013

2. Today’s Agenda Why Evaluation?!
Using a socio-ecological problem analysis framework to identify “causal pathways”
Developing strategies to address a problem based on upstream causes and risks
Developing a logic model for program planning and evaluation

3. Why Evaluation?! Root word: value Whose values?
Funders, clients, organization, etc.
It helps us tell our story
It helps us understand what we are doing well
and what we can improve…

4. “Traditional” Program Planning Cycle
Convene
Public Health/ Community Coalition
Assess Community /MCAH
Resources & Strengths/Capacity
Analyze Problem & Select Interventions
Evaluate/ Measure
Performance
Plan & Implement Programs
Develop
Objectives
Assess & Prioritize
Health Status/Problems
The information of these new paradigm impacts all the steps of the planning process. At step one in recognition of the importance of social determinants of health it makes sense to include a broader array of stakeholders that can assist in providing information on economic, social and policy issues that affect the MCAH population. During the assessment process data analysts will need to gather data at the individual, and community level on SES factors. But the main impact to the planning cycle will be during the analysis phase when the group is looking in depth at priority issues to develop strategies for change. It will also be critical at the intervention level to look at actions that can impact the higher level health determinants and also to consider act ivies that span the life course.

5. New Paradigms for Understanding Persistent Disparities
Emerging evidence: traditional clinical and public health efforts targeting individuals have not impacted health disparities
Multi-level analysis has demonstrated the significance of social determinants of health in explaining many of these disparities
Re-emergence of “life course model”, importance of prevention and intervention along age spectrum

6. Socio-Ecological Models

7. Lifecourse Model

8. Lifecourse Model as a Cycle
Mother overweight/diabetic
Poverty/fast food/food habits in young woman
Infant born LGA/Abnormal GT
Adolescent with increased # fat cells/
abnormal GT
Working Mom-Infant Fed
Formula
School with no PE, unsafe parks
latch key child watches TV /
gets more overweight
Family Culture - high fat, high sugar diet- child overweight
UCSF Family Health Outcomes Project

9. How do we Incorporate these New Paradigms into a Community/Public Health Planning Framework ?

10. African-American, Latino Children Drink More Sugary Fruit Juice Than Their White Peers
Share this story:
Share 
Print
By Juliana Bunim on May 30, 2013
While there has been a steep decline in kids’ consumption of sugar-sweetened beverages in California, African-American and Latino children may be replacing soda with 100 percent fruit juice while their white peers are not, according to a new study from UC San Francisco.
The study was the first to compare trends of sugar-sweetened beverages and 100 percent juice consumption in California.  
“The decrease in the consumption of sugar-sweetened beverages among kids is a promising public health trend,” said Amy Beck, MD, MPH, lead author and pediatrician at UCSF Benioff Children’s Hospital and San Francisco General Hospital and Trauma Center. “But it is concerning that minority children are increasing their consumption of 100 percent fruit juice, which often has just as much sugar as soda.”
…Fruit juice is available in schools through the National School Lunch Program, which provided low-cost or free lunches to more than 31 million children nationally each school day in Juice also is distributed to parents through the Women, Infants and Children (WIC) program, a federal grant program for low-income, nutritionally at-risk women and children up to age 5 and through the Child Care Food Program.

11. Seeing solutions with “Ecological Lenses”
At the individual level: Nutrition information (for parent or child)
At the family level: Nutrition education for parent and child
At the community level: a media campaign targeting families
At the organizational level: commit to soda AND juice free zones in community centers, provide access to fruit and clean water
At the policy level: change WIC policy to provide whole fruit and water filters, eliminate Juice vouchers for WIC OR
Eliminate fruit juice from subsidized school lunches; increase fruit and provide filling stations and water bottles in schools

12. Rationale for Doing a Formal Problem Analysis?
In order to identify effective intervention strategies it is necessary to understand the complex array of underlying factors that can impact a health outcome and how they relate to one another
Using a multilevel socio-ecological framework ensures that upstream factors are included
Relating upstream precursors to downstream outcomes forces us to explore the pathways by which upstream factors operate in a specific situation

13. Steps in a Problem Analysis
1. Examine epidemiologic data
2. Examine literature and consult experts (community and science)
3. Determine extent to which these factors are active in the community

14. Steps in a Problem Analysis
4. Determine relative contribution of each identified factor
5. Identify the interrelationships among factors – causal pathways
6. Determine the most effective points in the causal pathways for intervention

15. Psychological Factors
A Generic Framework for Health Problem Analysis
Social/Economic/Policy Level
SES
Safety
Education
Culture
Environment
Economy
Health Care Policies
Family/Community/Institutions Level
School/Workplace
Family/Household
Community
Health Care/Providers
Individual Level
Health Status/
Medical Conditions
Genetic/Biological
Psychological Factors
Cognitive Factors
Health Behaviors
Identified Problem

17. Too much screen time (TV computer) Insufficiency of physical activity
SES/Policy
Marketing by fast food industry
Unsafe neighborhood/streets
Inadequate subsidized health
Insurance for the poor
Poverty
City planning Policies that limit green space in some cities and neighborhoods
Lack of funding for education
Family/community
No opportunities for exercise in school or recreation areas
Family lack of health Insurance
Health services
lack of knowledge of nutrition
High costs for treatment
Families limited income
Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits)
Unavailability of healthy food choices in local stores , schools and restaurants
Too much screen time (TV computer)
Individual
Unhealthy diet
Insufficiency of physical activity
PROBLEM:
Childhood Obesity

18. Linkages precursors and problem Definition: The association between
Review the literature
Consult experts
Community
Science
Consult stakeholders
Analyze your data

19. Too much screen time (TV computer) Insufficient physical activity
SES/Policy
Marketing by fast food industry
Unsafe neighborhood/streets
Inadequate subsidized health
Insurance for the poor
Poverty
City planning Policies that limit green space in some cities and neighborhoods
Lack of funding for education
Family/community
No opportunities for exercise in school or recreation areas
Family lack of health Insurance
Health services
lack of knowledge of nutrition
High costs for treatment
Families limited income
Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits)
Unavailability of healthy food choices in local stores , schools and restaurants
Too much screen time (TV computer)
Individual
Unhealthy diet
Insufficient physical activity
PROBLEM:
Childhood Obesity

20. Diagramming Causal Pathways to Identify Strategic Interventions
Unsafe neighborhood streets
Intervention
Overweight
No opportunities for exercise in school or recreation areas
Say this is the pathway for low IZ rates. A survey of parents show they don’t perceive communicable childhood diseases to be a threat.
Where’s the intervention point?
What if you found out that two providers have poor tracking methods and don’t make reminder calls to parents -- what’s an intervention in that situation?
Tell Gerry’s story re: LBW in SF. Intensive programs of enhanced PNC. Didn’t make a dent in the outcomes at one clinic. Turned out, 65% of the pregnant women there smoked. Revised intervention to include smoking cessation.
Inadequate physical activity
Too much screen time (TV computer)

22. Gang Membership/Incarceration
Problem Analysis – Homey Clients
Social Determinants
 crime/gun policies/  prevention $
Economic policy Poverty / no jobs or bias in hiring
Racism/Destruction of
indigenous culture
 $ for schools/low standards/legacy of colonialism
Family/Community/Local Institutions
Police racial profiling
Community-lack of social capital/ cohesion/pride/trust
Family disintegration
Parents don’t care/use drug/don’t supervise/not educated/poor/lack cultural pride
 Jobs/lack of training opportunities
Barriers to accessing services:
Language/Cultural/ $
Transportation/Legal
Schools push kids out/racism in staff/poor quality/lack of cultural affinity
Individuals
Alcohol/drug use
Unsafe sex/ bad food/no exercise
Mental illness/PTSD/  self esteem/  resiliency/  trust/ sense of belonging
FAS/ADD/ADHD/other drug exposure
Learning difficulties/brain trauma/school failure/no college prep
PROBLEM:
Gang Membership/Incarceration

23. HOMEY- Kapuli Causal Pathway: Program Intervention Points
Communities:
Racism in schools
Racism by police
Lack of cultural identity in family/community
family disintegration
Kids:
Mental health problems
School failure
Low self esteem lack resiliency
No sense of true cultural identity
No sense of trust or belonging
Gangs/Jail
Society:
Racism
Lack of respect
for indigenous cultures
School failure
Self- Destructive Behaviors
Kapuli Interventions

24. ENVIRONMENTAL FACTORS
MOMEY-Kapuli Logic Model
Learning about history of social justice movements will give them tools for change
Learning about impact of racism/colonialism will give kids an undestanding of their own history
Episodes of tragedy due to racism in their community
Media/movie about examples of racism/harassment by police
ENVIRONMENTAL FACTORS
Street outreach
Educate about history of social justice movements for people of color/indigenous
10 strategies for change
Teach about racism/colonialism
Positive response during recap activity
Active participation in group
Successful school/CBO outreach
Schools
CBO’s
Kids from streets
Come back
Join political campaigns
Attend ceremonies
Express value of social action
Increase self esteem/resiliency/ capacity for action
Kids return to school
Kids get jobs
Kids stay out of gangs
Kids stay out of jail
Staff
Cameron
Space
Core Members
Funds
INPUTS
OUTPUTS
OUTCOMES – IMPACT
Activities
Participation
Short
Medium
Long-Term
ASSUMPTIONS:
Adapted from

25. Another Example: Obesity prevention/ intervention in the Bayview Hunters Point of SF
Designed in 2011, in partnership between:
UCSF
San Francisco Department of Public health
BVHP residents
other stakeholders
See handouts for problem analysis and logic models
As of 2013, focus is on:
Reducing consumption of sugar sweetened beverages
Activating community park space

26. Benefits
Clearly documents the decision making process for the group and for others
Can use simplified problem analysis or causal pathway diagram to communicate rationale for intervention strategy to policy makers and the public

27. Logic Model - Defined
Logic models are a common tool used by evaluators and program planners
Logic models are graphic depictions of the relationship between a program’s activities and its intended outcomes.
U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

28. Logic Model – Terms Relationship Intended
Logic models describe activities that comprise the program, the inter-relationship of those activities, and the link between these and outcomes.
Intended
Logic models depict “intended” outcomes of a program’s activities, rather than reality at any point in time. As the starting point for evaluation and planning, the model serves as an “outcomes roadmap” that shows the logic behind the program (describing why it will work). Of all the activities that might address this issue, these were chosen because they will work, and we have (or know) the resources needed . Over time, evaluation, research, and experience will help us learn what works, and the model will evolve.
U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

29. Logic Model - Components
Inputs
Resources that go into the program and are required to make it happen
Activities
events or actions done by the program and its staff
Outputs
Direct products of program activities, often measured in countable terms (e.g., #of zumba classes; # of participants).
Outcomes
The changes that result from the program’s activities and outputs, often expressed as short-term, intermediate, and long-term outcomes.

30. Basic Program Logic Model
U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

31. Let’s build a logic model!
Using the problem you’ve identified, begin by identifying the change you want to make (outcome of interest)
What is your “theory of change”? Or, what do you think it would take to make change? This will be the underlying logic that guides what activities your program does
What resources do you need (inputs)?

32. Q and A What was clarified for you; any “aha!” moments?
What still concerns or challenges you?

33. Contact Information
Community Engagement and Health Policy Program, CTSI/UCSF