Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluating Nutrition Education Programs Leslie A Lytle, PhD, RD Professor Division of Epidemiology and Community Health University of Minnesota.

Similar presentations


Presentation on theme: "Evaluating Nutrition Education Programs Leslie A Lytle, PhD, RD Professor Division of Epidemiology and Community Health University of Minnesota."— Presentation transcript:

1 Evaluating Nutrition Education Programs Leslie A Lytle, PhD, RD Professor Division of Epidemiology and Community Health University of Minnesota

2 l (Center for Youth Health Promotion website at UMN- tools, materials available for

3 The WINNERS!!! How to assess behavior Evaluating young children Assessing knowledge and attitudes Validity and reliability Sample size (unit of assignment) Strong study design

4 Other questions How to evaluate one-shot nutrition ed?/minumum exposure for nutr.ed to stick? Evaluating parental response to interventions targeted at the child (TEENS, CATCH examples) Home influence on kids eating choices Teacher evaluating change in kids…

5 Types of evaluation Outcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions.

6 Types of evaluation Impact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): Impact-which indicate how effective the intervention was in changing the target populations attitude, awareness and/or behavior. (knowledge, skills, attitudes, intention to act, behavior, or something else (pg 29 FTNEPG, 3/2007) Assumes that a causal relationship is being tested

7 Assessing behaviors, knowledge and attitudes What behaviors to focus on? How do knowledge and attitudes fit in? What tools are out there? Which behaviors are easier/more difficult to assess? How does age influence the evaluation tools that we use?

8 Recommended Behavioral Outcomes from FNS Eat fruits and vegetables Eat whole grains Consume fat-free or low-fat milk products every day Be physically active every day Balance caloric intake from food and beverages with calories expended

9 Recommended Behavioral Outcomes from FNS Evaluation tools should reflect knowledge, attitudes, behaviors related to these behavioral outcomes…

10 Designing an intervention and evaluation plan START with a conceptual model or logic model

11 Theory of Planned Behavior Attitude toward the behavior Subjectivenorm Perceivedbehavioralcontrol IntentionBehavior

12 Examining nutrition education impact/outcomes KNOWLEDGE ATTITUDES BEHAVIORS

13 Example of how KAB should line up in evaluation KNOWLEDGE: Does the student know how long is needed for adequate handwashing? ATTITUDES: Does the student believe that it is important to wash their hands? BEHAVIORS: Does the student report frequent handwashing or is observed consistently washing hands?

14 Examining nutrition education impact/outcomes KNOWLEDGE: Why versus How to knowledge How to KNOWLEDGE: Knowledge that will help one make healthier choices rather than knowledge for the sake of knowledge

15 Examining nutrition education impact/outcomes WHY KNOWLEDGE: Which of the following organs is responsible for secreting insulin? a. Kidneys b. Lungs c. Liver d. Pancreas

16 Examining nutrition education impact/outcomes HOW TO KNOWLEDGE (Example from Univ. of Nebraska): Which food is a lower fat snack? a. Pretzels b. Potato chips c. Doughnuts

17 Examining nutrition education impact/outcomes HOW TO KNOWLEDGE (Example from Univ. of Wyoming) Write the name of one food found in each food group from MyPyramid. Milk group Grain group Meat group Fruit group Vegetable group

18 More how to knowledge questions…(Kansas State) What is the healthiest snack choice? a.Soda pop and chips b.Milkshake and fries c.Fruit juice and pretzels Which foods would always be safe to pack in a sack lunch? a.Sliced ham b.Peanut Butter c.Sliced cheese How long should I wash my hands before I touch or eat food? a.As long as it takes to say my name b.As long as it takes to sing Happy Birthday c.As long as it takes to eat an apple

19 Assessing attitudes Motivation Beliefs, opinions Perceived benefits/barriers Subjective norm (how do others feel about this behavior?) Self efficacy (how confident do I feel about performing some behavior?

20 Attitude examples (Iowa State) I like to eat fruits and vegetables for snacks My friends like to eat fruits and vegetables for snacks How sure are you that you can fix fruits and vegetables to eat as snacks at home?

21

22 Measuring behavior Preference Self-report behavior Observed behavior

23

24 Other preference type questions See CATCH HBS: Section A: What food would you pick? Fruit and veggies lists 5-a-day Card Sorting Task for younger kids

25 Behavioral measures to assess Eat fruits and vegetables Eat whole grains Consume fat-free or low-fat milk products every day Be physically active every day Balance caloric intake from food and beverages with calories expended

26 Eat Fruits and Vegetables: Self report measures 24 hour recalls (valid in youth as young as 8 years old) Youth Adolescent Questionnaire-YAQ food frequency (valid in youth from about 6th grade and up) Validated 6 item fruit and veggie screener from BRFSS (valid in youth from about 6th grade and up)

27 Assessing fruit and veggie intake: 6th graders and older BRFSS tool These questions are about the foods you usually eat or drink. Please tell us how often, over the past year, you ate or drank each one. Mark the box for the response that best describes how often you eat or drink the food. Be sure to include foods you ate or drank at home, at school, at restaurants or anywhere else. 1. Over the past year, how many times did you drink fruit juices such as orange, grapefruit, or tomato? (Mark only ONE box.) Never or less than once per MONTH 1 time per DAY 1-3 times per MONTH 2 times per DAY 1-2 times per WEEK 3 times per DAY 3-4 times per WEEK 4 times per DAY 5-6 times per WEEK 5 or more times per DAY

28 2. Over the past year, how many times did you eat fruit (not counting juice)? Mark only ONE box.) Never or less than once per MONTH 1 time per DAY 1-3 times per MONTH 2 times per DAY 1-2 times per WEEK 3 times per DAY 3-4 times per WEEK 4 times per DAY 5-6 times per WEEK 5 or more times per DAY

29 Over the past year, how many times did you eat green salad? (Mark only one box.)

30 Over the past year, how many times did you eat potatoes (not including french fries, fried potatoes, or potato chips)?

31 Over the past year, how many times did you eat carrots? (Mark only ONE box.)

32 Over the past year, how many times did you eat vegetables (not counting carrots, potatoes, or salad)? (Mark only ONE box)

33 Eat Fruits and Vegetables: Self report measures How about for kids younger than 8? –Observation –Parental reporting

34 Behavioral outcomes to assess Fruits and vegetables Whole grain consumption: Are there valid and reliable tools? Can youth identify whole grain foods? Fat free or low fat dairy Be physically active every day Balance caloric intake from food and beverages with calories expended

35 Behavioral outcomes to assess Fruits and vegetables Whole grain consumption Fat free or low fat dairy: EAT Survey 10-item scale Be physically active every day Balance caloric intake from food and beverages with calories expended

36 Assessing Calcium intakes in middle school age youth Project EAT (Lisa Harnack, P.I.) 10 item scale Reliability and validity assessed Reliability: test:retest one week apart Validity: 3 24-hour recalls as criterion measure Harnack, Lytle, Story et al, JADA 2006;106:

37 Reliability and Validity Reliability: will the instrument produce the same result if applied twice? Do respondents understand the question in a consistent manner? (test-retest) Are observers consistent in documenting what they see? (inter-rater reliability) Do the items that I have included to assess a construct relate to each other as I would expect them to? (cronbachs alpha)

38 Reliability and Validity Criterion Validity: How well does a measure compare with a gold standard or criterion measure? Face Validity: Would others reading the item understand the concept that I am trying to assess? Does the measure appearto measure the desired construct? Content validity: Does my instrument capture all the elements that I think are important Construct validity: Is the measure related to other measures as one would expect?

39 Assessing Calcium intakes in middle school age youth Test-retest correlations: Reliability SampleICC Total sample0.74 Females0.79 Males y.o y.o.0.75

40 Assessing Calcium intakes in middle school age youth Assessing Validity: Comparing calcium intake using 3 methods (n=248) 24 hour recalls: 993mg (499) 10 item CA FFQ: 856mg (570) 1 YRBS question: 423mg (344) Correlations –Recalls and FFQ= 0.43 –Recalls and YRBS= 0.37

41 Behavioral outcomes to assess Fruits and vegetables Whole grain consumption Fat free or low fat dairy Be physically active every day Balance caloric intake from food and beverages with calories expended

42 TEENS Survey Questions Do you get some regular physical activity outside of school? By regular we mean at least three times a week for at least 20 minutes at a time. Most of the time Usually Once in a while Hardly ever Never When you think about how hard you work out when you are physically active, do you find that you are Breathing much harder than usual Breathing somewhat harder than usual Breathing only a little harder than usual Breathing the same as usual

43 How many hours per day do you usually watch TV during the weekdays? I dont watch TV during the weekdays Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day How many hours per day do you usually watch TV during the weekend? I dont watch TV during the weekend Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day

44 How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekdays? I dont play video games during the weekdays Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekend? I dont play video games during the weekend Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day

45 Behavioral outcomes to assess Fruits and vegetables Whole grain consumption Fat free or low fat dairy Be physically active every day Balance caloric intake from food and beverages with calories expended

46 Assessing caloric intake and expenditure is very difficult Related behavioral messages and outcomes to assess on the energy intake side may include: –Consumption of a healthy portion size –Eating meals and few snacks –Conscious eating –Reduction of empty calories –Reduction of soft drinks

47 Balance caloric intake from food and beverages with calories expended Assessing caloric intake and expenditure is very difficult Related behavioral messages and outcomes to assess on the energy expenditure side include: –Reducing television time –Increased walking or biking for transportation –Small increases in activity every day –More leisure time activity

48 Balance caloric intake from food and beverages with calories expended: Examples On a typical day, how many hours do you watch TV, view videos or work/play on the computer? (Iowa) –I dont watch tv, etc.. –Less than one hour –1 hour/day –2 hours/day I eat breakfast in the morning (UMN) –NeverSometimesAlmost AlwaysAlways

49 Energy balance knowledge: Examples from IDEA Question % correct YouthParent If someone sits all day, they do Not need to eat any calories(F)78%98% Alcohol contains calories (T)77%99% The sweetner used in Gatorade and Other soft drinks is healthier than the Sweetener used in sodas (F)47%69% Most youth who are active need to Consume sports drinks to replace Electrolytes and minerals (F)40%72%

50 How do assess young children (less than 7 years old or 2nd grade and younger) Very challenging Cannot think abstractly, limited reading and writing skills Will be able to perform simple knowledge tests Use observation assessment tools as possible, conducted by objective observers, to assess behavior

51 Consider outcomes that do not rely on self report Plate waste (Colorado) Visually Monitoring Consumption Take versus Eat

52 Visually monitoring consumption Purpose: To develop and evaluate a simple observation method to assess student consumption of specific menu items offered in school cafeterias Developed the Visual Food Monitoring Form Selected 4 menu items to estimate consumption End of tray line, dots on trays with items (established our denominator) At tray return area, staff completed the VFMF Snyder, Fee, Lytle, Hann, 1996: School Food Service Research Review, 20(2),

53

54 Results Evaluation staff made 3 visits to 4 schools Observed 1839 trays Conducted pre-post evaluations of modified menu items Results: –81% ate at least 3/4 of higher-fat entrées –78% ate at least 3/4 or lower-fat modifications

55 Can food taken be used as a proxy for food eaten? Purpose: If we record what foods are on students trays at the end of the cafeteria line, can that be used as a proxy for foods consumed? Rationale: Observing and recording actual consumption is labor-intensive and expensive, (approximately $40/child observed!) Gray, Lytle, Mays, et al, 2002; JADA 102:

56 Can food taken be used as a proxy for food eaten? Validation study: n=350 5th grade students Recorded foods present on trays at end of tray line Recorded foods actually consumed Used different staff to record taken and eaten data. Results: –Correlations of taken:eaten fruits= 0.59 –Correlations of taken: eaten veggies=0.51 Gray, Lytle, Mays, et al, 2002; JADA 102:

57

58 Types of evaluation Outcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions. Study design: Post-only with a control/comparison group Pre-post

59 Pre-post study design What are some of the problems with this design?

60 Pre-post study design What are some of the problems with this design? –Was the change REALLY caused by the intervention? Some other thing happened? Response bias? Did they learn the right answer?

61 Pre-post study design What are some options? –Find a comparison/control group –Do a delayed intervention condition –Use different interventions in different classes and use one evaluation tool that assesses behavioral objectives from both intervention

62 Do you need to conduct an evaluation every time you run an intervention?

63 Sample study design Baseline measures 20 Schools 10 schools intervention 10 school waiting list 1st follow up eval 10 schools post-treament 10 school intervention Final follow- up measures M 1/2 I M M R=randomizationM=measurement period R

64 Impact evaluation Impact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): Impact-which indicate how effective the intervention was in changing the target populations attitude, awareness and/or behavior. (knowledge, skills, attitudes, intention to act, behavior, or something else (pg 29 FTNEPG, 3/2007) Assumes that a causal relationship is being tested Study design: Randomized control/comparison design

65 Establishing a causal relationship What is meant by a causal relationship? A --- B If we introduce A, then B is more likely to occur than if A was NOT introduced –B doesnt ALWAYS occur with A –Sometimes B occurs even without A –There is some assumption of temporality (A precedes B)

66 What are the best study designs to evaluate IMPACT? Have at least 2 time points Clearly identify what A is Clearly identify what B is Anticipate potential confounding factors and assess them Have a control or comparison group Have an adequate sample

67 Control versus Comparison group Control groups are randomized Comparison groups are NOT randomized

68 Gold standard for assessing impact Randomized Control Trial –Unit of assignment is unit of randomization –Clustering that naturally occurs in groups included in the analysis plan (random effects) –Sample size calculations are based on the unit of assignment/randomization –Baseline measures, randomization, intervention, follow-up measures –Can be very expensive

69 Randomized control trial with attention paid to unit of assignment What is Unit of Assignment?

70 Unit of assignment and the FNS document Page 7: Describe the unit of assignment to intervention and control groups Describe how assignment to intervention and control groups was carried out Describe how many units and individuals were in the intervention and control groups at the start of the intervention.

71 Unit of Assignment At what UNIT is the intervention delivered? At what UNIT does randomization into condition occur? –Out-patient nutrition counseling: Individual –Community trial: An entire community –FNS nutrition programs: A classroom, a school, a school district, an afterschool program

72 What is the Unit of Assignment when you… Train all teachers in a school to deliver an intervention? Work with the school food service director at the district office to change offerings in school cafeterias? Recruit youth in an after school program to participate in a special nutrition education class

73 TEENS as an example 16 schools were randomized into control or treatment condition School is the unit of assignment and analysis Over 3500 students in 16 schools completed surveys The sample size of TEENS = 16

74 Unit of assignment Consider a classroom or a school as the unit of assignment One classroom/school receives an intervention One classroom/school does not The mean is the aggregated scores of all kids in the classroom/school The analysis compares the means between the units of assignment

75 School or classroom becomes the unity of assignment and analysis May be able to add other elements of the intervention to the model The analysis model needs to include a term for the clustering of kids within the unit

76 What are some potential problems in using a classroom as the unit of assignment and then comparing students in control and intervention classrooms?

77 Keep in mind Focus on specific behaviors Attempt to focus your intervention (content, time, type of delivery)- too much flexibility is very difficult to evaluate Make sure your intervention objectives, intervention strategies, and KAB measures are all lined up

78 Keep in mind Look for valid and reliable instruments in the literature Observation may be a cost-effective way to assess change-especially with young kids Keep it simple Move beyond pre-post study designs Look for a control/comparison group

79 `

80 Why evaluate? Fulfill an obligation or regulatory requirement Document how your program is being implemented Document how your program is being received Document if your program is working Provide feedback for future program planning

81 Types of evaluation: Qualitative and quantitative evaluation Qualitative: –Open ended questions –Focus groups/key informant interviews/observations –Allows for stories to be told –Allows for respondents to describe their own experiences or opinions

82 Types of evaluation: Qualitative and quantitative evaluation Quantitative: –Closed ended questions –Response can be QUANTIFIED/COUNTED –Surveys/questionnaires/observations –Allows for presentation of data in a tabular format –Allows for statistical comparison –Respondents provide responses to specific questions

83 Types of evaluation Qualitative: EXAMPLE Interview teachers prior to the development of a new set of classroom materials. Ask them the following question: Tell me about your experiences in teaching nutrition education in the classroom. What are some things that you enjoy about teaching nutrition education to your children? What are some things that make it a challenge?

84 Types of evaluation Quantitative: EXAMPLE Ask teachers to fill out a survey that includes the following question: Which of the following make teaching nutrition education challenging? Check all that apply: –Not enough classroom time –Available materials are not culturally appropriate –Available materials are not age appropriate –No available materials –Lack of support for nutrition education from administration –Lack of cooking/food preparation facilities –Lack of interest by students –Inadequate background in nutrition

85 When might you use QUALITATIVE EVALUATION? When might you use QUANTITATIVE EVALUATION?

86 Types of evaluation Formative Process Impact Outcome

87 Types of evaluation Formative: Conducting information for the intent of FORMING or developing a program or for FORMING or adapting a program

88 Process Evaluation Part of formative evaluation Provides documentation of what is going on in a program, how a program is being implemented and received by a variety of stakeholders

89 Formative evaluation: Examples Conducting focus groups with teachers and youth to see what types of nutrition programs that they are interested in Conducting a survey to document parents beliefs about the school food environment Conducting key informant interviews after the completion of a pilot program to see what teachers think needs to be changed for the next phase of programming

90 Process Evaluation May provide documentation of: –Completeness of implementation: Is all of the intervention being implemented as planned? –Fidelity to the intervention plan: Is the intervention being implemented as planned? –Dose of the intervention: How much of the intervention is being received by intended parties? –Reach of the intervention: How many of the intended participants are receiving the program? –How do stakeholders feel about the program?

91 Kansas State: Process Evaluation Examples Do you plan to spend more time on nutrition education in your classroom than you did prior to this nutrition education program? How was class response to the program? What part(s) of the program did students enjoy the most? Would you be interested in having the program again next year? Do you have any requests, suggestions, ideas for improvements, or other comments?

92

93

94

95 But isnt it all about the environment? Do we have a chance to positively influence behaviors when the environment is toxic?

96 An Ecological Model Davison-Birch, 2001

97


Download ppt "Evaluating Nutrition Education Programs Leslie A Lytle, PhD, RD Professor Division of Epidemiology and Community Health University of Minnesota."

Similar presentations


Ads by Google