Presentation is loading. Please wait.

Presentation is loading. Please wait.

Frameworks for program planning and evaluation: Problem Analysis and Logic Models Geraldine Oliva M.D., MPH Roberto Ariel Vargas, MPH James Rouse-Iñiguez,

Similar presentations


Presentation on theme: "Frameworks for program planning and evaluation: Problem Analysis and Logic Models Geraldine Oliva M.D., MPH Roberto Ariel Vargas, MPH James Rouse-Iñiguez,"— Presentation transcript:

1 Frameworks for program planning and evaluation: Problem Analysis and Logic Models Geraldine Oliva M.D., MPH Roberto Ariel Vargas, MPH James Rouse-Iñiguez, MA Paula Fleisher, MA June 2013 June 2013

2 Today’s Agenda Why Evaluation?! Why Evaluation?! Using a socio-ecological problem analysis framework to identify “causal pathways” Using a socio-ecological problem analysis framework to identify “causal pathways” Developing strategies to address a problem based on upstream causes and risks Developing strategies to address a problem based on upstream causes and risks Developing a logic model for program planning and evaluation Developing a logic model for program planning and evaluation

3 Why Evaluation?! Root word: value Root word: value Whose values? Funders, clients, organization, etc. It helps us tell our story It helps us tell our story It helps us understand what we are doing well It helps us understand what we are doing well and what we can improve…

4 “Traditional” Program Planning Cycle Convene Public Health/ Community Coalition Assess Community /MCAH Resources & Strengths/Capacity Analyze Problem & Select Interventions Evaluate/ Measure Performance Plan & Implement Programs Develop Objectives Assess & Prioritize Health Status/Problems

5 New Paradigms for Understanding Persistent Disparities Emerging evidence: traditional clinical and public health efforts targeting individuals have not impacted health disparities Emerging evidence: traditional clinical and public health efforts targeting individuals have not impacted health disparities Multi-level analysis has demonstrated the significance of social determinants of health in explaining many of these disparities Multi-level analysis has demonstrated the significance of social determinants of health in explaining many of these disparities Re-emergence of “life course model”, importance of prevention and intervention along age spectrum Re-emergence of “life course model”, importance of prevention and intervention along age spectrum

6 Socio-Ecological Models

7 Lifecourse Model

8 UCSF Family Health Outcomes Project Mother overweight/diabetic Working Mom-Infant Fed Formula Family Culture - high fat, high sugar diet- child overweight Adolescent with increased # fat cells/ abnormal GT School with no PE, unsafe parks latch key child watches TV / gets more overweight Lifecourse Model as a Cycle Poverty/fast food/food habits in young woman Infant born LGA/Abnormal GT

9 How do we Incorporate these New Paradigms into a Community/Public Health Planning Framework ?

10 African-American, Latino Children Drink More Sugary Fruit Juice Than Their White Peers Share this story: Share Print By Juliana Bunim on May 30, 2013 Juliana Bunim While there has been a steep decline in kids ’ consumption of sugar- sweetened beverages in California, African-American and Latino children may be replacing soda with 100 percent fruit juice while their white peers are not, according to a new study from UC San Francisco. The study was the first to compare trends of sugar-sweetened beverages and 100 percent juice consumption in California. “ The decrease in the consumption of sugar-sweetened beverages among kids is a promising public health trend, ” said Amy Beck, MD, MPH, lead author and pediatrician at UCSF Benioff Children ’ s Hospital and San Francisco General Hospital and Trauma Center. “ But it is concerning that minority children are increasing their consumption of 100 percent fruit juice, which often has just as much sugar as soda. ” Amy Beck …Fruit juice is available in schools through the National School Lunch Program, which provided low-cost or free lunches to more than 31 million children nationally each school day in Juice also is distributed to parents through the Women, Infants and Children (WIC) program, a federal grant program for low-income, nutritionally at-risk women and children up to age 5 and through the Child Care Food Program.

11 Seeing solutions with “Ecological Lenses” At the individual level: Nutrition information (for parent or child) At the family level: Nutrition education for parent and child At the community level: a media campaign targeting families At the organizational level: commit to soda AND juice free zones in community centers, provide access to fruit and clean water At the policy level: change WIC policy to provide whole fruit and water filters, eliminate Juice vouchers for WIC OR Eliminate fruit juice from subsidized school lunches; increase fruit and provide filling stations and water bottles in schools

12 Rationale for Doing a Formal Problem Analysis? In order to identify effective intervention strategies it is necessary to understand the complex array of underlying factors that can impact a health outcome and how they relate to one another In order to identify effective intervention strategies it is necessary to understand the complex array of underlying factors that can impact a health outcome and how they relate to one another Using a multilevel socio-ecological framework ensures that upstream factors are included Using a multilevel socio-ecological framework ensures that upstream factors are included Relating upstream precursors to downstream outcomes forces us to explore the pathways by which upstream factors operate in a specific situation Relating upstream precursors to downstream outcomes forces us to explore the pathways by which upstream factors operate in a specific situation

13 Steps in a Problem Analysis 1.Examine epidemiologic data 2.Examine literature and consult experts (community and science) 3.Determine extent to which these factors are active in the community

14 Steps in a Problem Analysis 4.Determine relative contribution of each identified factor 5.Identify the interrelationships among factors – causal pathways 6.Determine the most effective points in the causal pathways for intervention

15 SES Social/Economic/Policy Level Environment Safety A Generic Framework for Health Problem Analysis Education Economy Health Care Policies Family/Community/Institutions Level Family/Household Community School/Workplace Health Care/Providers Individual Level Genetic/Biological Psychological Factors Health Status/ Medical Conditions Cognitive Factors Health Behaviors Identified Problem Culture

16

17 PROBLEM: Family/community SES/Policy Individual Unavailability of healthy food choices in local stores, schools and restaurants Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits) Families limited income Unhealthy diet Inadequate subsidized health Insurance for the poor Poverty City planning Policies that limit green space in some cities and neighborhoods Childhood Obesity Health services lack of knowledge of nutrition High costs for treatment No opportunities for exercise in school or recreation areas Insufficiency of physical activity Unsafe neighborhood/streets Marketing by fast food industry Family lack of health Insurance Lack of funding for education Too much screen time (TV computer)

18 Linkages Definition: The association between precursors and problem  Review the literature  Consult experts  Community  Science  Consult stakeholders  Analyze your data

19 PROBLEM: Family/community SES/Policy Individual Unavailability of healthy food choices in local stores, schools and restaurants Parental knowledge and behavior ( time with kids, food prep, own eating and exercise habits) Families limited income Unhealthy diet Inadequate subsidized health Insurance for the poor Poverty City planning Policies that limit green space in some cities and neighborhoods Childhood Obesity Health services lack of knowledge of nutrition High costs for treatment No opportunities for exercise in school or recreation areas Insufficient physical activity Unsafe neighborhood/streets Marketing by fast food industry Family lack of health Insurance Lack of funding for education Too much screen time (TV computer)

20 Diagramming Causal Pathways to Identify Strategic Interventions Unsafe neighborhood streets No opportunities for exercise in school or recreation areas Too much screen time (TV computer) Inadequate physical activity Overweight Intervention

21

22 PROBLEM: Family/Community/Local Institutions Social Determinants Individuals Schools push kids out/racism in staff/poor quality/lack of cultural affinity Barriers to accessing services: Language/Cultural/ $ Transportation/Legal Mental illness/PTSD/  self esteem/  resiliency/  trust/ sense of belonging Racism/Destruction of indigenous culture Economic policy Poverty / no jobs or bias in hiring  crime/gun policies/  prevention $ Gang Membership/Incarceration Community-lack of social capital/ cohesion/pride/trust Alcohol/drug use Unsafe sex/ bad food/no exercise  $ for schools/low standards/legacy of colonialism Problem Analysis – Homey Clients Family disintegration Parents don’t care/use drug/don’t supervise/not educated/poor/lack cultural pride FAS/ADD/ADH D/other drug exposure Learning difficulties/brain trauma/school failure/no college prep  Jobs/lack of training opportunities Police racial profiling

23 HOMEY- Kapuli Causal Pathway: Program Intervention Points Society: Racism Lack of respect for indigenous cultures Communities: Racism in schools Racism by police Lack of cultural identity in family/community family disintegration Kids: Mental health problems School failure Low self esteem lack resiliency No sense of true cultural identity No sense of trust or belonging Gangs/Jail School failure Self- Destructive Behaviors Kapuli Interventions

24 Learning about history of social justice movements will give them tools for change Learning about impact of racism/colonialism will give kids an undestanding of their own history Episodes of tragedy due to racism in their community Media/movie about examples of racism/harassment by police ENVIRONMENTAL FACTORS Street outreach Educate about history of social justice movements for people of color/indigenous 10 strategies for change Teach about racism/colonialism Positive response during recap activity Active participation in group Successful school/CBO outreach Schools CBO’s Kids from streets Come back Join political campaigns Attend ceremonies Express value of social action Increase self esteem/resiliency/ capacity for action Kids return to school Kids get jobs Kids stay out of gangs Kids stay out of jail Staff Cameron Space Core Members Funds INPUTS OUTPUTS OUTCOMES – IMPACT ActivitiesParticipationShortMediumLong-Term ASSUMPTIONS: Adapted from MOMEY-Kapuli Logic Model

25 Another Example : Obesity prevention/ intervention in the Bayview Hunters Point of SF Designed in 2011, in partnership between: UCSF San Francisco Department of Public health BVHP residents other stakeholders See handouts for problem analysis and logic models As of 2013, focus is on : Reducing consumption of sugar sweetened beverages Activating community park space

26 Benefits Clearly documents the decision making process for the group and for others Clearly documents the decision making process for the group and for others Can use simplified problem analysis or causal pathway diagram to communicate rationale for intervention strategy to policy makers and the public Can use simplified problem analysis or causal pathway diagram to communicate rationale for intervention strategy to policy makers and the public

27 Logic Model - Defined Logic models are a common tool used by evaluators and program planners Logic models are a common tool used by evaluators and program planners Logic models are graphic depictions of the relationship between a program’s activities and its intended outcomes. Logic models are graphic depictions of the relationship between a program’s activities and its intended outcomes. U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

28 Logic Model – Terms Relationship Relationship Logic models describe activities that comprise the program, the inter-relationship of those activities, and the link between these and outcomes.Logic models describe activities that comprise the program, the inter-relationship of those activities, and the link between these and outcomes. Intended Intended Logic models depict “intended” outcomes of a program’s activities, rather than reality at any point in time. As the starting point for evaluation and planning, the model serves as an “outcomes roadmap” that shows the logic behind the program (describing why it will work). Of all the activities that might address this issue, these were chosen because they will work, and we have (or know) the resources needed. Over time, evaluation, research, and experience will help us learn what works, and the model will evolve.Logic models depict “intended” outcomes of a program’s activities, rather than reality at any point in time. As the starting point for evaluation and planning, the model serves as an “outcomes roadmap” that shows the logic behind the program (describing why it will work). Of all the activities that might address this issue, these were chosen because they will work, and we have (or know) the resources needed. Over time, evaluation, research, and experience will help us learn what works, and the model will evolve. U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

29 Logic Model - Components Inputs Inputs Resources that go into the program and are required to make it happenResources that go into the program and are required to make it happen Activities Activities events or actions done by the program and its staffevents or actions done by the program and its staff Outputs Outputs Direct products of program activities, often measured in countable terms (e.g., #of zumba classes; # of participants).Direct products of program activities, often measured in countable terms (e.g., #of zumba classes; # of participants). Outcomes Outcomes The changes that result from the program’s activities and outputs, often expressed as short-term, intermediate, and long-term outcomes.The changes that result from the program’s activities and outputs, often expressed as short-term, intermediate, and long-term outcomes.

30 Basic Program Logic Model U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. Introduction to program evaluation for public health programs: A self-study guide. Atlanta, GA: Centers for Disease Control and Prevention, 2005, p. 25.

31 Let’s build a logic model! Using the problem you’ve identified, begin by identifying the change you want to make (outcome of interest) Using the problem you’ve identified, begin by identifying the change you want to make (outcome of interest) What is your “theory of change”? Or, what do you think it would take to make change? This will be the underlying logic that guides what activities your program does What is your “theory of change”? Or, what do you think it would take to make change? This will be the underlying logic that guides what activities your program does What resources do you need (inputs)? What resources do you need (inputs)?

32 Q and A What was clarified for you; any “aha!” moments? What was clarified for you; any “aha!” moments? What still concerns or challenges you? What still concerns or challenges you?

33 Contact Information Community Engagement and Health Policy Program, CTSI/UCSF


Download ppt "Frameworks for program planning and evaluation: Problem Analysis and Logic Models Geraldine Oliva M.D., MPH Roberto Ariel Vargas, MPH James Rouse-Iñiguez,"

Similar presentations


Ads by Google