Evaluating Nutrition Education Programs Leslie A Lytle, PhD, RD Professor Division of Epidemiology and Community Health University of Minnesota

l Lytle@epi.umn.edu Cyhp@epi.umn.edu (Center for Youth Health Promotion website at UMN- tools, materials available for free)

The WINNERS!!! How to assess behavior Evaluating young children Assessing knowledge and attitudes Validity and reliability Sample size (unit of assignment) Strong study design

Other questions How to evaluate “one-shot” nutrition ed?/minumum exposure for nutr.ed to stick? Evaluating parental response to interventions targeted at the child (TEENS, CATCH examples) Home influence on kids’ eating choices Teacher evaluating change in kids…

Types of evaluation Outcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): “Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions.”

Types of evaluation Impact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): Impact-which indicate how effective the intervention was in changing the target population’s attitude, awareness and/or behavior. (“knowledge, skills, attitudes, intention to act, behavior, or something else” (pg 29 FTNEPG, 3/2007) Assumes that a causal relationship is being tested

Assessing behaviors, knowledge and attitudes What behaviors to focus on? How do knowledge and attitudes fit in? What tools are out there? Which behaviors are easier/more difficult to assess? How does age influence the evaluation tools that we use?

Recommended Behavioral Outcomes from FNS Eat fruits and vegetables Eat whole grains Consume fat-free or low-fat milk products every day Be physically active every day Balance caloric intake from food and beverages with calories expended

Recommended Behavioral Outcomes from FNS Evaluation tools should reflect knowledge, attitudes, behaviors related to these behavioral outcomes…

Designing an intervention and evaluation plan START with a conceptual model or logic model

Theory of Planned Behavior Attitude toward the behavior Intention Behavior Subjective norm Perceived behavioral control

Examining nutrition education impact/outcomes KNOWLEDGE  ATTITUDES BEHAVIORS

Example of how KAB should line up in evaluation KNOWLEDGE: Does the student know how long is needed for adequate handwashing? ATTITUDES: Does the student believe that it is important to wash their hands? BEHAVIORS: Does the student report frequent handwashing or is observed consistently washing hands?

Examining nutrition education impact/outcomes KNOWLEDGE: “Why” versus “How to knowledge” “How to” KNOWLEDGE: Knowledge that will help one make healthier choices rather than knowledge for the sake of knowledge

Examining nutrition education impact/outcomes WHY KNOWLEDGE: • Which of the following organs is responsible for secreting insulin? a. Kidneys b. Lungs c. Liver d. Pancreas

Examining nutrition education impact/outcomes HOW TO KNOWLEDGE (Example from Univ. of Nebraska): • Which food is a lower fat snack? a. Pretzels b. Potato chips c. Doughnuts

Examining nutrition education impact/outcomes HOW TO KNOWLEDGE (Example from Univ. of Wyoming) • Write the name of one food found in each food group from MyPyramid. Milk group Grain group Meat group Fruit group Vegetable group

More “how to” knowledge questions…(Kansas State) What is the healthiest snack choice? Soda pop and chips Milkshake and fries Fruit juice and pretzels Which foods would always be safe to pack in a sack lunch? Sliced ham Peanut Butter Sliced cheese How long should I wash my hands before I touch or eat food? As long as it takes to say my name As long as it takes to sing “Happy Birthday” As long as it takes to eat an apple

Assessing attitudes Motivation Beliefs, opinions Perceived benefits/barriers Subjective norm (how do others feel about this behavior?) Self efficacy (how confident do I feel about performing some behavior?

Attitude examples (Iowa State) I like to eat fruits and vegetables for snacks My friends like to eat fruits and vegetables for snacks How sure are you that you can fix fruits and vegetables to eat as snacks at home?

Measuring behavior Preference Self-report behavior Observed behavior

Other preference type questions See CATCH HBS: Section A: What food would you pick? Fruit and veggies lists 5-a-day Card Sorting Task for younger kids

Behavioral measures to assess Eat fruits and vegetables Eat whole grains Consume fat-free or low-fat milk products every day Be physically active every day Balance caloric intake from food and beverages with calories expended

Eat Fruits and Vegetables: Self report measures 24 hour recalls (valid in youth as young as 8 years old) Youth Adolescent Questionnaire-YAQ food frequency (valid in youth from about 6th grade and up) Validated 6 item fruit and veggie screener from BRFSS (valid in youth from about 6th grade and up)

Assessing fruit and veggie intake: 6th graders and older BRFSS tool These questions are about the foods you usually eat or drink. Please tell us how often, over the past year, you ate or drank each one. Mark the box for the response that best describes how often you eat or drink the food. Be sure to include foods you ate or drank at home, at school, at restaurants or anywhere else. 1. Over the past year, how many times did you drink fruit juices such as orange, grapefruit, or tomato? (Mark only ONE box.) oNever or less than once per MONTH o1 time per DAY o1-3 times per MONTH o2 times per DAY o1-2 times per WEEK o3 times per DAY o3-4 times per WEEK o4 times per DAY o5-6 times per WEEK o5 or more times per DAY

2. Over the past year, how many times did you eat fruit (not counting juice)? Mark only ONE box.) oNever or less than once per MONTH o1 time per DAY o1-3 times per MONTH o2 times per DAY o1-2 times per WEEK o3 times per DAY o3-4 times per WEEK o4 times per DAY o5-6 times per WEEK o5 or more times per DAY

Over the past year, how many times did you eat green salad Over the past year, how many times did you eat green salad? (Mark only one box.)

Over the past year, how many times did you eat potatoes (not including french fries, fried potatoes, or potato chips)?

Over the past year, how many times did you eat carrots Over the past year, how many times did you eat carrots? (Mark only ONE box.)

Over the past year, how many times did you eat vegetables (not counting carrots, potatoes, or salad)? (Mark only ONE box)

Eat Fruits and Vegetables: Self report measures How about for kids younger than 8? Observation Parental reporting

Behavioral outcomes to assess Fruits and vegetables Whole grain consumption: Are there valid and reliable tools? Can youth identify whole grain foods? Fat free or low fat dairy Be physically active every day Balance caloric intake from food and beverages with calories expended

Behavioral outcomes to assess Fruits and vegetables Whole grain consumption Fat free or low fat dairy: EAT Survey 10-item scale Be physically active every day Balance caloric intake from food and beverages with calories expended

Assessing Calcium intakes in middle school age youth Project EAT (Lisa Harnack, P.I.) 10 item scale Reliability and validity assessed Reliability: test:retest one week apart Validity: 3 24-hour recalls as criterion measure Harnack, Lytle, Story et al, JADA 2006;106: 1790-1795

Reliability and Validity Reliability: will the instrument produce the same result if applied twice? Do respondents understand the question in a consistent manner? (test-retest) Are observers consistent in documenting what they see? (inter-rater reliability) Do the items that I have included to assess a construct relate to each other as I would expect them to? (cronbach’s alpha)

Reliability and Validity Criterion Validity: How well does a measure compare with a “gold standard” or criterion measure? Face Validity: Would others reading the item understand the concept that I am trying to assess? Does the measure “appear”to measure the desired construct? Content validity: Does my instrument capture all the elements that I think are important Construct validity: Is the measure related to other measures as one would expect?

Assessing Calcium intakes in middle school age youth Test-retest correlations: Reliability Sample ICC Total sample 0.74 Females 0.79 Males 0.66 11-12 y.o 0.74 13-14 y.o. 0.75

Assessing Calcium intakes in middle school age youth Assessing Validity: Comparing calcium intake using 3 methods (n=248) 24 hour recalls: 993mg (499) 10 item CA FFQ: 856mg (570) 1 YRBS question: 423mg (344) Correlations Recalls and FFQ= 0.43 Recalls and YRBS= 0.37

Behavioral outcomes to assess Fruits and vegetables Whole grain consumption Fat free or low fat dairy Be physically active every day Balance caloric intake from food and beverages with calories expended

TEENS Survey Questions Do you get some regular physical activity outside of school? By regular we mean at least three times a week for at least 20 minutes at a time. Most of the time Usually Once in a while Hardly ever Never When you think about how hard you work out when you are physically active, do you find that you are Breathing much harder than usual Breathing somewhat harder than usual Breathing only a little harder than usual Breathing the same as usual

How many hours per day do you usually watch TV during the weekdays? I don’t watch TV during the weekdays Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day How many hours per day do you usually watch TV during the weekend? I don’t watch TV during the weekend

How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekdays? I don’t play video games during the weekdays Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekend? I don’t play video games during the weekend

Behavioral outcomes to assess Fruits and vegetables Whole grain consumption Fat free or low fat dairy Be physically active every day Balance caloric intake from food and beverages with calories expended

Balance caloric intake from food and beverages with calories expended Assessing caloric intake and expenditure is very difficult Related behavioral messages and outcomes to assess on the energy intake side may include: Consumption of a healthy portion size Eating meals and few snacks Conscious eating Reduction of empty calories Reduction of soft drinks

Balance caloric intake from food and beverages with calories expended Assessing caloric intake and expenditure is very difficult Related behavioral messages and outcomes to assess on the energy expenditure side include: Reducing television time Increased walking or biking for transportation Small increases in activity every day More leisure time activity

Balance caloric intake from food and beverages with calories expended: Examples On a typical day, how many hours do you watch TV, view videos or work/play on the computer? (Iowa) I don’t watch tv, etc.. Less than one hour 1 hour/day 2 hours/day I eat breakfast in the morning (UMN) Never Sometimes Almost Always Always

Energy balance knowledge: Examples from IDEA Question % correct Youth Parent If someone sits all day, they do Not need to eat any calories (F) 78% 98% Alcohol contains calories (T) 77% 99% The sweetner used in Gatorade and Other soft drinks is healthier than the Sweetener used in sodas (F) 47% 69% Most youth who are active need to Consume sports drinks to replace Electrolytes and minerals (F) 40% 72%

How do assess young children (less than 7 years old or 2nd grade and younger) Very challenging Cannot think abstractly, limited reading and writing skills Will be able to perform simple knowledge tests Use observation assessment tools as possible, conducted by objective observers, to assess behavior

Consider outcomes that do not rely on self report Plate waste (Colorado) Visually Monitoring Consumption Take versus Eat

Visually monitoring consumption Purpose: To develop and evaluate a simple observation method to assess student consumption of specific menu items offered in school cafeterias Developed the Visual Food Monitoring Form Selected 4 menu items to estimate consumption End of tray line, dots on trays with items (established our denominator) At tray return area, staff completed the VFMF Snyder, Fee, Lytle, Hann, 1996: School Food Service Research Review, 20(2), 63-68.

Results Evaluation staff made 3 visits to 4 schools Observed 1839 trays Conducted pre-post evaluations of modified menu items Results: 81% ate at least 3/4 of higher-fat entrées 78% ate at least 3/4 or lower-fat modifications

Can food taken be used as a proxy for food eaten? Purpose: If we record what foods are on students’ trays at the end of the cafeteria line, can that be used as a proxy for foods consumed? Rationale: Observing and recording actual consumption is labor-intensive and expensive, (approximately $40/child observed!) Gray, Lytle, Mays, et al, 2002; JADA 102:407-409.

Can food taken be used as a proxy for food eaten? Validation study: n=350 5th grade students Recorded foods present on trays at end of tray line Recorded foods actually consumed Used different staff to record taken and eaten data. Results: Correlations of taken:eaten fruits= 0.59 Correlations of taken: eaten veggies=0.51 Gray, Lytle, Mays, et al, 2002; JADA 102:407-409.

Types of evaluation Outcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): “Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions.” Study design: Post-only with a control/comparison group Pre-post

Pre-post study design What are some of the problems with this design?

Pre-post study design What are some of the problems with this design? Was the change REALLY caused by the intervention? Some other thing happened? Response bias? Did they learn the “right answer?

Pre-post study design What are some options? Find a comparison/control group Do a delayed intervention condition Use different interventions in different classes and use one evaluation tool that assesses behavioral objectives from both intervention

Do you need to conduct an evaluation every time you run an intervention?

Sample study design M 1/2 I M 1/2 I M 10 schools post-treament 10 schools intervention 1st follow up eval Baseline measures 20 Schools Final follow-up measures R 10 school waiting list 1st follow up eval 10 school intervention M 1/2 I M 1/2 I M R=randomization M=measurement period

Impact evaluation Impact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007): Impact-which indicate how effective the intervention was in changing the target population’s attitude, awareness and/or behavior. (“knowledge, skills, attitudes, intention to act, behavior, or something else” (pg 29 FTNEPG, 3/2007) Assumes that a causal relationship is being tested Study design: Randomized control/comparison design

Establishing a causal relationship What is meant by a causal relationship? A --- B If we introduce A, then B is more likely to occur than if A was NOT introduced B doesn’t ALWAYS occur with A Sometimes B occurs even without A There is some assumption of temporality (A precedes B)

What are the best study designs to evaluate IMPACT? Have at least 2 time points Clearly identify what A is Clearly identify what B is Anticipate potential confounding factors and assess them Have a control or comparison group Have an adequate sample

Control versus Comparison group Control groups are randomized Comparison groups are NOT randomized

Gold standard for assessing impact Randomized Control Trial Unit of assignment is unit of randomization Clustering that naturally occurs in groups included in the analysis plan (random effects) Sample size calculations are based on the unit of assignment/randomization Baseline measures, randomization, intervention, follow-up measures Can be very expensive

Randomized control trial with attention paid to unit of assignment What is “Unit of Assignment”?

Unit of assignment and the FNS document Page 7: “Describe the unit of assignment to intervention and control groups” “Describe how assignment to intervention and control groups was carried out” “Describe how many units and individuals were in the intervention and control groups at the start of the intervention.”

Unit of Assignment At what UNIT is the intervention delivered? At what UNIT does randomization into condition occur? Out-patient nutrition counseling: Individual Community trial: An entire community FNS nutrition programs: A classroom, a school, a school district, an afterschool program

What is the Unit of Assignment when you… Train all teachers in a school to deliver an intervention? Work with the school food service director at the district office to change offerings in school cafeterias? Recruit youth in an after school program to participate in a special nutrition education class

TEENS as an example 16 schools were randomized into control or treatment condition School is the unit of assignment and analysis Over 3500 students in 16 schools completed surveys The sample size of TEENS = 16

Unit of assignment Consider a classroom or a school as the unit of assignment One classroom/school receives an intervention One classroom/school does not The mean is the aggregated scores of all kids in the classroom/school The analysis compares the means between the units of assignment

School or classroom becomes the unity of assignment and analysis May be able to add other elements of the intervention to the model The analysis model needs to include a term for the clustering of kids within the unit

What are some potential problems in using a classroom as the unit of assignment and then comparing students in “control” and “intervention” classrooms?

Keep in mind Focus on specific behaviors Attempt to focus your intervention (content, time, type of delivery)- too much flexibility is very difficult to evaluate Make sure your intervention objectives, intervention strategies, and KAB measures are all lined up

Keep in mind Look for valid and reliable instruments in the literature Observation may be a cost-effective way to assess change-especially with young kids Keep it simple Move beyond pre-post study designs Look for a control/comparison group

`

Why evaluate? Fulfill an obligation or regulatory requirement Document how your program is being implemented Document how your program is being received Document if your program is working Provide feedback for future program planning

Types of evaluation: Qualitative and quantitative evaluation Open ended questions Focus groups/key informant interviews/observations Allows for stories to be told Allows for respondents to describe their own experiences or opinions

Types of evaluation: Qualitative and quantitative evaluation Closed ended questions Response can be QUANTIFIED/COUNTED Surveys/questionnaires/observations Allows for presentation of data in a tabular format Allows for statistical comparison Respondents provide responses to specific questions

Types of evaluation Qualitative: EXAMPLE Interview teachers prior to the development of a new set of classroom materials. Ask them the following question: “Tell me about your experiences in teaching nutrition education in the classroom. What are some things that you enjoy about teaching nutrition education to your children? What are some things that make it a challenge?”

Types of evaluation Quantitative: EXAMPLE Ask teachers to fill out a survey that includes the following question: Which of the following make teaching nutrition education challenging? Check all that apply: Not enough classroom time Available materials are not culturally appropriate Available materials are not age appropriate No available materials Lack of support for nutrition education from administration Lack of cooking/food preparation facilities Lack of interest by students Inadequate background in nutrition

When might you use QUALITATIVE EVALUATION? When might you use QUANTITATIVE EVALUATION?

Types of evaluation Formative Process Impact Outcome

Types of evaluation Formative: Conducting information for the intent of FORMING or developing a program or for FORMING or adapting a program

Process Evaluation Part of formative evaluation Provides documentation of what is going on in a program, how a program is being implemented and received by a variety of stakeholders

Formative evaluation: Examples Conducting focus groups with teachers and youth to see what types of nutrition programs that they are interested in Conducting a survey to document parents’ beliefs about the school food environment Conducting key informant interviews after the completion of a pilot program to see what teachers think needs to be changed for the next phase of programming

Process Evaluation May provide documentation of: Completeness of implementation: Is all of the intervention being implemented as planned? Fidelity to the intervention plan: Is the intervention being implemented as planned? Dose of the intervention: How much of the intervention is being received by intended parties? Reach of the intervention: How many of the intended participants are receiving the program? How do stakeholders feel about the program?

Kansas State: Process Evaluation Examples Do you plan to spend more time on nutrition education in your classroom than you did prior to this nutrition education program? How was class response to the program? What part(s) of the program did students enjoy the most? Would you be interested in having the program again next year? Do you have any requests, suggestions, ideas for improvements, or other comments?

But isn’t it all about the environment? Do we have a chance to positively influence behaviors when the environment is toxic?

An Ecological Model Davison-Birch, 2001