1. MPR Impact Reporting Survey questions, aggregation, and moving forward
Sandra Biller
May 10, 2018
SNAP-Ed Regional Conference

2. Benefits of Aggregation
Show region-wide impacts
Better illustrate program value to stakeholders
Assess impacts across states/agencies
Highlight programs of excellence
Monitor performance
Quality improvement
Track progress towards outcomes at a regional level

3. Background State/regional impact reports Can aggregate reach
Cannot easily aggregate impacts
Problems with aggregation
Known variations in measurement of impacts
Lack of knowledge about specifics
What tools are being used?
What questions are being asked?
How are surveys conducted? (pre/post, retrospective, etc.)
Can we collect this information to get a baseline?

4. Objective & Methods Objective
Assess similarities and differences in survey strategies
Priority indicators MT1-3
Adults and youth
Goal: inform potential aggregation solutions used in future fiscal years
Methods
Collect surveys, questions, or data points used from Implementing or State Agencies
All 10 states within MPR contacted
Data compiled and examined for themes, with a focus on:
Fruit Consumption
Vegetable Consumption
Food Resource Management
Physical Activity

5. Results: Overview 80% of states responded (n=8)
84% of agencies included (n=11)
Data is still incomplete
257 Questions
119 Adult
138 Youth
All states track both adults and youth
90% of questions are used by states

6. Adult Fruit Consumption
Example Fruit Frequency Questions
Source
# of States
Do you eat more than one kind of fruit each day?
EFNEP Adult Behavior Change Checklist 4
Eat different kind of fruits
Kansas 1
How often do you typically eat fruit?
Cooking Matters Adults, Parents, Families Survey
During the past week, not counting juice, how often did you eat fruit?
Create Farm Fresh Food
During the past week, how often did you eat more than one type of fruit?
Food $ense (6-lesson follow-up)

7. Adult Vegetable Consumption
Example Vegetable Frequency Questions
Source
# of States
Do you eat more than one kind of vegetable each day?
EFNEP Adult Behavior Change Checklist 4
Eat different kinds of vegetables
Kansas 1
During the past week, how often did you eat more than one type of vegetable?
Food $ense (6-lesson follow-up)
I eat more than 1 kind of vegetable per day because of the parent wellness workshops.
Culture of Wellness in Preschools (post)

8. Adult Fruit & Vegetable Highlights
Two main strategies
Measure frequency of intake
Measure volume of intake
More variation for vegetable consumption
Several states use EFNEP questions
Wording of questions is similar
Examples:
How often do you typically eat fruit?
During the past week, not counting juice, how often did you eat fruit?
During the past week, how often did you eat more than one type of fruit?

9. Adult Food Resource Management
Example Food Resource Management Questions by Category
Source
# of States
MT2b - How often do you use the Nutrition Facts label to make food choices?
EFNEP Adult Behavior Change Checklist, Cooking Matters Adults, Parents, Families Survey 7
MT2g - How often do you run out of food before the end of the month?
EFNEP Adult Behavior Change Checklist 5
MT2h - How often do you compare prices before you buy food? 6
MT2j - How often do you shop with a grocery list?

10. Adult Food Resource Management Highlights
100% of responding states track two indicators
MT2b – Read Nutrition Facts labels
MT2j – Shop with a list
EFNEP questions used by a large majority
Wording of questions is very similar:
Examples:
How often do you shop with a grocery list?
How often do you use a grocery list when grocery shopping?

11. Adult Physical Activity
Example Physical Activity Questions
Source
# of States
Do you participate in at least 30 minutes of moderate physical activity daily?
EFNEP Adult Behavior Change Checklist 3
Are you active on 4 or more days a week? 2
I participate in physical activities for at least 30 minutes per day
Kansas 1
Choose to be physically active for at least 30 minutes, 5 times a week
Food $ense (6-month)
How often are you physically active for at least 30 minutes on 4 or more days of the week?
UNL-NEP
No. of days physically active at least 60 mins/day
Culture of Wellness in Preschools

12. Adult Physical Activity Highlights
88% of responding states track increase in time spent physically active
Some states specify moderate physical activity – would bring this to 100%
Less agreement on specific questions
Wording of questions is similar
Examples:
Do you participate in at least 30 minutes of moderate physical activity daily?
I participate in physical activities for at least 30 minutes per day
Choose to be physically active for at least 30 minutes, 5 times a week

13. Youth Significant variation in evaluation:
Grades/ages targeted
Questions asked
Methodology
Pre/Post
Post-only
Parent/staff surveys
Little overlap between states
When overlap did occur, an age- specific EFNEP survey was used
Due to these challenges, was unable to pull out many common themes.

14. Challenges and Limitations
Variations in methodology, particularly with youth
Some disagreement on what questions count for what indicators
Example: Youth FRM
ST vs. MT
In some cases, questions asked are not similar
Examples:
Do you eat more than one kind of vegetable each day?
How often do you typically eat green salad?
More common for youth
New and upcoming changes in surveys (EFNEP)
Tight time frame for collecting data
Did not collect question response options (i.e., what type of Likert scale was used?)

15. Commonalities(Adult)
All states track MT1-3
Significant overlap in outcome measures
Within outcome measures, similar wording of questions
Large variability in specific questions/survey tools

16. How should we aggregate our data?
Align outcome measures
Example: MT1c. Ate more than one kind of fruit throughout the day or week
Ask the same questions with the same response options
Example: Do you eat more than one kind of fruit each day?
Use the same evaluation tool
Benefits and challenges to data aggregation
SNAP-Ed adoption of the Interpretive Guide – a first step in the process
More discussion needed to decide on a solution

17. Many Thanks
All states who so generously let me use their data for this project!
MPR Evaluation workgroup
Wyoming SNAP-Ed Team