Presentation on theme: "Evaluating Nutrition Education Programs"— Presentation transcript:
1 Evaluating Nutrition Education Programs Leslie A Lytle, PhD, RDProfessorDivision of Epidemiology and Community HealthUniversity of Minnesota
2 l(Center for Youth Health Promotion website at UMN- tools, materials available for free)
3 The WINNERS!!! How to assess behavior Evaluating young children Assessing knowledge and attitudesValidity and reliabilitySample size (unit of assignment)Strong study design
4 Other questionsHow to evaluate “one-shot” nutrition ed?/minumum exposure for nutr.ed to stick?Evaluating parental response to interventions targeted at the child (TEENS, CATCH examples)Home influence on kids’ eating choicesTeacher evaluating change in kids…
5 Types of evaluationOutcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007):“Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions.”
6 Types of evaluationImpact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007):Impact-which indicate how effective the intervention was in changing the target population’s attitude, awareness and/or behavior. (“knowledge, skills, attitudes, intention to act, behavior, or something else” (pg 29 FTNEPG, 3/2007)Assumes that a causal relationship is being tested
7 Assessing behaviors, knowledge and attitudes What behaviors to focus on?How do knowledge and attitudes fit in?What tools are out there?Which behaviors are easier/more difficult to assess?How does age influence the evaluation tools that we use?
8 Recommended Behavioral Outcomes from FNS Eat fruits and vegetablesEat whole grainsConsume fat-free or low-fat milk products every dayBe physically active every dayBalance caloric intake from food and beverages with calories expended
9 Recommended Behavioral Outcomes from FNS Evaluation tools should reflect knowledge, attitudes, behaviors related to these behavioral outcomes…
10 Designing an intervention and evaluation plan START with a conceptual model or logic model
11 Theory of Planned Behavior Attitudetoward thebehaviorIntentionBehaviorSubjectivenormPerceivedbehavioralcontrol
13 Example of how KAB should line up in evaluation KNOWLEDGE: Does the student know how long is needed for adequate handwashing?ATTITUDES: Does the student believe that it is important to wash their hands?BEHAVIORS: Does the student report frequent handwashing or is observed consistently washing hands?
14 Examining nutrition education impact/outcomes KNOWLEDGE:“Why” versus “How to knowledge”“How to” KNOWLEDGE:Knowledge that will help one make healthier choices rather than knowledge for the sake of knowledge
15 Examining nutrition education impact/outcomes WHY KNOWLEDGE:• Which of the following organs is responsible for secreting insulin?a. Kidneysb. Lungsc. Liverd. Pancreas
16 Examining nutrition education impact/outcomes HOW TO KNOWLEDGE (Example from Univ. of Nebraska):• Which food is a lower fat snack?a. Pretzelsb. Potato chipsc. Doughnuts
17 Examining nutrition education impact/outcomes HOW TO KNOWLEDGE (Example from Univ. of Wyoming)• Write the name of one food found in each food group from MyPyramid.Milk groupGrain groupMeat groupFruit groupVegetable group
18 More “how to” knowledge questions…(Kansas State) What is the healthiest snack choice?Soda pop and chipsMilkshake and friesFruit juice and pretzelsWhich foods would always be safe to pack in a sack lunch?Sliced hamPeanut ButterSliced cheeseHow long should I wash my hands before I touch or eat food?As long as it takes to say my nameAs long as it takes to sing “Happy Birthday”As long as it takes to eat an apple
19 Assessing attitudes Motivation Beliefs, opinions Perceived benefits/barriersSubjective norm (how do others feel about this behavior?)Self efficacy (how confident do I feel about performing some behavior?
20 Attitude examples (Iowa State) I like to eat fruits and vegetables for snacksMy friends like to eat fruits and vegetables for snacksHow sure are you that you can fix fruits and vegetables to eat as snacks at home?
24 Other preference type questions See CATCH HBS: Section A: What food would you pick?Fruit and veggies lists5-a-day Card Sorting Task for younger kids
25 Behavioral measures to assess Eat fruits and vegetablesEat whole grainsConsume fat-free or low-fat milk products every dayBe physically active every dayBalance caloric intake from food and beverages with calories expended
26 Eat Fruits and Vegetables: Self report measures 24 hour recalls (valid in youth as young as 8 years old)Youth Adolescent Questionnaire-YAQ food frequency (valid in youth from about 6th grade and up)Validated 6 item fruit and veggie screener from BRFSS (valid in youth from about 6th grade and up)
27 Assessing fruit and veggie intake: 6th graders and older BRFSS tool These questions are about the foods you usually eat or drink. Please tell us how often, over the past year, you ate or drank each one. Mark the box for the response that best describes how often you eat or drink the food. Be sure to include foods you ate or drank at home, at school, at restaurants or anywhere else.1. Over the past year, how many times did you drink fruit juices such as orange, grapefruit, or tomato? (Mark only ONE box.)oNever or less than once per MONTHo1 time per DAYo1-3 times per MONTHo2 times per DAYo1-2 times per WEEKo3 times per DAYo3-4 times per WEEKo4 times per DAYo5-6 times per WEEKo5 or more times per DAY
28 2. Over the past year, how many times did you eat fruit (not counting juice)? Mark only ONE box.) oNever or less than once per MONTHo1 time per DAYo1-3 times per MONTHo2 times per DAYo1-2 times per WEEKo3 times per DAYo3-4 times per WEEKo4 times per DAYo5-6 times per WEEKo5 or more times per DAY
29 Over the past year, how many times did you eat green salad Over the past year, how many times did you eat green salad? (Mark only one box.)
30 Over the past year, how many times did you eat potatoes (not including french fries, fried potatoes, or potato chips)?
31 Over the past year, how many times did you eat carrots Over the past year, how many times did you eat carrots? (Mark only ONE box.)
32 Over the past year, how many times did you eat vegetables (not counting carrots, potatoes, or salad)? (Mark only ONE box)
33 Eat Fruits and Vegetables: Self report measures How about for kids younger than 8?ObservationParental reporting
34 Behavioral outcomes to assess Fruits and vegetablesWhole grain consumption: Are there valid and reliable tools? Can youth identify whole grain foods?Fat free or low fat dairyBe physically active every dayBalance caloric intake from food and beverages with calories expended
35 Behavioral outcomes to assess Fruits and vegetablesWhole grain consumptionFat free or low fat dairy: EAT Survey 10-item scaleBe physically active every dayBalance caloric intake from food and beverages with calories expended
36 Assessing Calcium intakes in middle school age youth Project EAT (Lisa Harnack, P.I.)10 item scaleReliability and validity assessedReliability: test:retest one week apartValidity: hour recalls as criterion measureHarnack, Lytle, Story et al, JADA 2006;106:
37 Reliability and Validity Reliability: will the instrument produce the same result if applied twice?Do respondents understand the question in a consistent manner? (test-retest)Are observers consistent in documenting what they see? (inter-rater reliability)Do the items that I have included to assess a construct relate to each other as I would expect them to? (cronbach’s alpha)
38 Reliability and Validity Criterion Validity: How well does a measure compare with a “gold standard” or criterion measure?Face Validity: Would others reading the item understand the concept that I am trying to assess? Does the measure “appear”to measure the desired construct?Content validity: Does my instrument capture all the elements that I think are importantConstruct validity: Is the measure related to other measures as one would expect?
39 Assessing Calcium intakes in middle school age youth Test-retest correlations: ReliabilitySample ICCTotal sample 0.74FemalesMales11-12 y.o13-14 y.o
40 Assessing Calcium intakes in middle school age youth Assessing Validity: Comparing calcium intake using 3 methods (n=248)24 hour recalls: 993mg (499)10 item CA FFQ: 856mg (570)1 YRBS question: 423mg (344)CorrelationsRecalls and FFQ= 0.43Recalls and YRBS= 0.37
41 Behavioral outcomes to assess Fruits and vegetablesWhole grain consumptionFat free or low fat dairyBe physically active every dayBalance caloric intake from food and beverages with calories expended
42 TEENS Survey Questions Do you get some regular physical activity outside of school? By regular we mean at least three times a week for at least 20 minutes at a time.Most of the timeUsuallyOnce in a whileHardly everNeverWhen you think about how hard you work out when you are physically active, do you find that you areBreathing much harder than usualBreathing somewhat harder than usualBreathing only a little harder than usualBreathing the same as usual
43 How many hours per day do you usually watch TV during the weekdays? I don’t watch TV during the weekdaysLess than 1 hour per day1-2 hours per day3-4 hours per dayMore than 4 hours per dayHow many hours per day do you usually watch TV during the weekend?I don’t watch TV during the weekend
44 How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekdays?I don’t play video games during the weekdaysLess than 1 hour per day1-2 hours per day3-4 hours per dayMore than 4 hours per dayHow many hours per day do you usually play video games (including hand-held video games and computer games) during the weekend?I don’t play video games during the weekend
45 Behavioral outcomes to assess Fruits and vegetablesWhole grain consumptionFat free or low fat dairyBe physically active every dayBalance caloric intake from food and beverages with calories expended
46 Balance caloric intake from food and beverages with calories expended Assessing caloric intake and expenditure is very difficultRelated behavioral messages and outcomes to assess on the energy intake side may include:Consumption of a healthy portion sizeEating meals and few snacksConscious eatingReduction of empty caloriesReduction of soft drinks
47 Balance caloric intake from food and beverages with calories expended Assessing caloric intake and expenditure is very difficultRelated behavioral messages and outcomes to assess on the energy expenditure side include:Reducing television timeIncreased walking or biking for transportationSmall increases in activity every dayMore leisure time activity
48 Balance caloric intake from food and beverages with calories expended: Examples On a typical day, how many hours do you watch TV, view videos or work/play on the computer? (Iowa)I don’t watch tv, etc..Less than one hour1 hour/day2 hours/dayI eat breakfast in the morning (UMN)Never Sometimes Almost Always Always
49 Energy balance knowledge: Examples from IDEA Question % correctYouth ParentIf someone sits all day, they doNot need to eat any calories (F) 78% 98%Alcohol contains calories (T) 77% 99%The sweetner used in Gatorade andOther soft drinks is healthier than theSweetener used in sodas (F) 47% 69%Most youth who are active need toConsume sports drinks to replaceElectrolytes and minerals (F) 40% 72%
50 How do assess young children (less than 7 years old or 2nd grade and younger) Very challengingCannot think abstractly, limited reading and writing skillsWill be able to perform simple knowledge testsUse observation assessment tools as possible, conducted by objective observers, to assess behavior
51 Consider outcomes that do not rely on self report Plate waste (Colorado)Visually Monitoring ConsumptionTake versus Eat
52 Visually monitoring consumption Purpose: To develop and evaluate a simple observation method to assess student consumption of specific menu items offered in school cafeteriasDeveloped the Visual Food Monitoring FormSelected 4 menu items to estimate consumptionEnd of tray line, dots on trays with items (established our denominator)At tray return area, staff completed the VFMFSnyder, Fee, Lytle, Hann, 1996: School Food Service Research Review, 20(2),
54 Results Evaluation staff made 3 visits to 4 schools Observed 1839 traysConducted pre-post evaluations of modified menu itemsResults:81% ate at least 3/4 of higher-fat entrées78% ate at least 3/4 or lower-fat modifications
55 Can food taken be used as a proxy for food eaten? Purpose: If we record what foods are on students’ trays at the end of the cafeteria line, can that be used as a proxy for foods consumed?Rationale: Observing and recording actual consumption is labor-intensive and expensive, (approximately $40/child observed!)Gray, Lytle, Mays, et al, 2002; JADA 102:
56 Can food taken be used as a proxy for food eaten? Validation study: n=350 5th grade studentsRecorded foods present on trays at end of tray lineRecorded foods actually consumedUsed different staff to record taken and eaten data.Results:Correlations of taken:eaten fruits= 0.59Correlations of taken: eaten veggies=0.51Gray, Lytle, Mays, et al, 2002; JADA 102:
58 Types of evaluationOutcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007):“Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions.”Study design: Post-only with a control/comparison groupPre-post
59 Pre-post study designWhat are some of the problems with this design?
60 Pre-post study design What are some of the problems with this design? Was the change REALLY caused by the intervention?Some other thing happened?Response bias? Did they learn the “right answer?
61 Pre-post study design What are some options? Find a comparison/control groupDo a delayed intervention conditionUse different interventions in different classes and use one evaluation tool that assesses behavioral objectives from both intervention
62 Do you need to conduct an evaluation every time you run an intervention?
63 Sample study design M 1/2 I M 1/2 I M 10 schools post-treament 10 schools intervention1st follow up evalBaselinemeasures20 SchoolsFinal follow-up measuresR10 school waiting list1st follow up eval10 school interventionM1/2 IM1/2 IMR=randomizationM=measurement period
64 Impact evaluationImpact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007):Impact-which indicate how effective the intervention was in changing the target population’s attitude, awareness and/or behavior. (“knowledge, skills, attitudes, intention to act, behavior, or something else” (pg 29 FTNEPG, 3/2007)Assumes that a causal relationship is being testedStudy design: Randomized control/comparison design
65 Establishing a causal relationship What is meant by a causal relationship?A --- BIf we introduce A, then B is more likely to occur than if A was NOT introducedB doesn’t ALWAYS occur with ASometimes B occurs even without AThere is some assumption of temporality (A precedes B)
66 What are the best study designs to evaluate IMPACT? Have at least 2 time pointsClearly identify what A isClearly identify what B isAnticipate potential confounding factors and assess themHave a control or comparison groupHave an adequate sample
67 Control versus Comparison group Control groups are randomizedComparison groups are NOT randomized
68 Gold standard for assessing impact Randomized Control TrialUnit of assignment is unit of randomizationClustering that naturally occurs in groups included in the analysis plan (random effects)Sample size calculations are based on the unit of assignment/randomizationBaseline measures, randomization, intervention, follow-up measuresCan be very expensive
69 Randomized control trial with attention paid to unit of assignment What is “Unit of Assignment”?
70 Unit of assignment and the FNS document Page 7:“Describe the unit of assignment to intervention and control groups”“Describe how assignment to intervention and control groups was carried out”“Describe how many units and individuals were in the intervention and control groups at the start of the intervention.”
71 Unit of AssignmentAt what UNIT is the intervention delivered? At what UNIT does randomization into condition occur?Out-patient nutrition counseling: IndividualCommunity trial: An entire communityFNS nutrition programs: A classroom, a school, a school district, an afterschool program
72 What is the Unit of Assignment when you… Train all teachers in a school to deliver an intervention?Work with the school food service director at the district office to change offerings in school cafeterias?Recruit youth in an after school program to participate in a special nutrition education class
73 TEENS as an example16 schools were randomized into control or treatment conditionSchool is the unit of assignment and analysisOver 3500 students in 16 schools completed surveysThe sample size of TEENS = 16
74 Unit of assignmentConsider a classroom or a school as the unit of assignmentOne classroom/school receives an interventionOne classroom/school does notThe mean is the aggregated scores of all kids in the classroom/schoolThe analysis compares the means between the units of assignment
75 School or classroom becomes the unity of assignment and analysis May be able to add other elements of the intervention to the modelThe analysis model needs to include a term for the clustering of kids within the unit
76 What are some potential problems in using a classroom as the unit of assignment and then comparing students in “control” and “intervention” classrooms?
77 Keep in mind Focus on specific behaviors Attempt to focus your intervention (content, time, type of delivery)- too much flexibility is very difficult to evaluateMake sure your intervention objectives, intervention strategies, and KAB measures are all lined up
78 Keep in mind Look for valid and reliable instruments in the literature Observation may be a cost-effective way to assess change-especially with young kidsKeep it simpleMove beyond pre-post study designsLook for a control/comparison group
80 Why evaluate? Fulfill an obligation or regulatory requirement Document how your program is being implementedDocument how your program is being receivedDocument if your program is workingProvide feedback for future program planning
81 Types of evaluation: Qualitative and quantitative evaluation Open ended questionsFocus groups/key informant interviews/observationsAllows for stories to be toldAllows for respondents to describe their own experiences or opinions
82 Types of evaluation: Qualitative and quantitative evaluation Closed ended questionsResponse can be QUANTIFIED/COUNTEDSurveys/questionnaires/observationsAllows for presentation of data in a tabular formatAllows for statistical comparisonRespondents provide responses to specific questions
83 Types of evaluation Qualitative: EXAMPLE Interview teachers prior to the development of a new set of classroom materials. Ask them the following question:“Tell me about your experiences in teaching nutrition education in the classroom. What are some things that you enjoy about teaching nutrition education to your children? What are some things that make it a challenge?”
84 Types of evaluation Quantitative: EXAMPLE Ask teachers to fill out a survey that includes the following question:Which of the following make teaching nutrition education challenging? Check all that apply:Not enough classroom timeAvailable materials are not culturally appropriateAvailable materials are not age appropriateNo available materialsLack of support for nutrition education from administrationLack of cooking/food preparation facilitiesLack of interest by studentsInadequate background in nutrition
85 When might you use QUALITATIVE EVALUATION? When might you use QUANTITATIVE EVALUATION?
86 Types of evaluationFormativeProcessImpactOutcome
87 Types of evaluationFormative: Conducting information for the intent of FORMING or developing a program or for FORMING or adapting a program
88 Process Evaluation Part of formative evaluation Provides documentation of what is going on in a program, how a program is being implemented and received by a variety of stakeholders
89 Formative evaluation: Examples Conducting focus groups with teachers and youth to see what types of nutrition programs that they are interested inConducting a survey to document parents’ beliefs about the school food environmentConducting key informant interviews after the completion of a pilot program to see what teachers think needs to be changed for the next phase of programming
90 Process Evaluation May provide documentation of: Completeness of implementation: Is all of the intervention being implemented as planned?Fidelity to the intervention plan: Is the intervention being implemented as planned?Dose of the intervention: How much of the intervention is being received by intended parties?Reach of the intervention: How many of the intended participants are receiving the program?How do stakeholders feel about the program?
91 Kansas State: Process Evaluation Examples Do you plan to spend more time on nutrition education in your classroom than you did prior to this nutrition education program?How was class response to the program?What part(s) of the program did students enjoy the most?Would you be interested in having the program again next year?Do you have any requests, suggestions, ideas for improvements, or other comments?