1. Western Region Evaluation Framework Update
Andy Riesenberg, MSPH
Team Lead, Food Security and Obesity Prevention
Supplemental Nutrition Assistance Program
Western Region

2. Feb 2015

3. Introduction

4. Community Example: Linda Vista Neighborhood, San Diego
Feb 2015

5. What sets SNAP-Ed apart?
Focus on low-income population
Comprehensive strategies (nutrition education, marketing, environmental changes)
Community and stakeholder engagement
Evidence-based prevention strategies
Feb 2015

6. Evaluation Framework – At a Glance
Feb 2015

7. The outcomes are presented in the short‐term; medium‐term; long‐term; and impacts. Shortterm
outcomes are early markers of program success. While there is no set time‐parameter for
short‐term activities, generally these are the immediate results that can be measured during or
after program implementation. Medium‐term and long‐term outcomes build upon previous
accomplishments. For some outcomes in the outer spheres of the SEM, it will be difficult, if not
impossible, to tease out the relative contributions of SNAP‐Ed. For these measures, we might
consider the collective impact of partnerships.
At the individual level, the distinction between medium-term and long-term is that medium-term represents intermediate markers of progression toward meeting the DGA and Physical Activity Guidelines for American (PAGA) recommendations. Medium-term outcomes represent changes in actions or behaviors as measured by pre- and post-surveys before and after individual and group based education and health promotion activities. The medium-term indicators are actionable for on-going program evaluation.
Long-Term indicators at the individual level are more in alignment with the DGA and PAGA. The long-term indicators are designed for use by States that either conduct surveys among SNAP-Ed participants, or States that conduct annual or biennial population-level surveillance of low-income audiences using state-run surveillance systems, such as the Behavioral Risk Factor Surveillance System (BRFSS). The reference points for cups or servings of foods and beverages in long-term indicators align with population benchmarks used in surveillance surveys. This approach facilitates comparisons between the SNAP-Ed population and the general population when measuring State or national trends in nutrition and physical activity.
Feb 2015

8. Evaluation Questions
Individual‐level To what extent does SNAP‐Ed programming improve participants’ diet, physical activity, and health? Environmental‐level To what extent does SNAP-Ed programming facilitate access and create appeal for improved dietary and physical activity choices in the settings where nutrition education is provided? Sectors of Influence To what extent is the SNAP-Ed grant program integrated into comprehensive strategies that collectively impact lifelong healthy eating and active living in low-income communities? Social and Cultural Norms and Values To what extent do community-level obesity prevention strategies impact the public’s priorities, lifestyle choices, and values for healthy living?
Feb 2015

9. Healthy People in Healthy Places in Healthy Communities
Individual:
People
Environmental:
Places
Sectors of Influence:
Communities
Unit of Analysis
Youth and Adults
Settings and Networks
Low-income Communities
Reach
Participants
Organizations and Places
Populations and Jurisdictions
What Gets Measured?
Behavioral changes in nutrition, physical activity
Adoption, Implementation, Effectiveness, Maintenance
Collective Impact/
Nutrition in all policies
How to Measure?
Surveys
Focus Groups/ Interviews
Direct Observation
Assessments/ Environmental Scans
Document Review
Network Analysis
Data warehouses
CDC/DHHS/RWJF Indicators
Health registries
Feb 2015

10. Example
Feb 2015

11. FY ‘15 Priority (Preferred) Indicators
Feb 2015

12. FY 15 Priority Indicators – Basic
Individual Level
Environmental Level
Sectors of Influence
MT1: MyPlate Behaviors
ST4: Identification of Opportunities
MT9: Agriculture
MT2: Shopping Behaviors
ST6: Partnerships
MT3: Physical Activity Behaviors
LT2: Fruits and Vegetables
LT4: Dairy
LT6: Food Security
Feb 2015

13. FY 15 Priority Indicators – Enhanced (States > $2 mil)
Environmental Level
Sectors of Influence
MT4: Nutrition Supports Adopted
MT8: Local Government
MT5: Physical Activity Supports Adopted
MT12: Health Care
MT6: Marketing and Messaging
Feb 2015

14. Measuring Individual Changes
Feb 2015

15. Number of SNAP-Ed participants with a matched pre- and post-test.
MT2 Shopping Behaviors 
Logic Model Component
Medium-Term Outcome –
Changes in individual and family behaviors that reflect smarter shopping and food resource management strategies, enabling participants to stretch their food resource dollars.
What to measure:
Number, or %, of SNAP-Ed participants who increased targeted shopping behaviors during the period assessed. For each participant, compare responses to survey administered before the first lesson and after the last lesson.
Numerator:
Number of SNAP-Ed participants with a matched pre- and post-test who performed the behavior at post-test for the first time or with greater intensity or frequency.
Denominator:
Number of SNAP-Ed participants with a matched pre- and post-test.
Examples of behaviors
Read nutrition facts or nutrition ingredients lists
Compare prices before buying foods
Identify foods on sale or use coupons
Shop with a list
How to Measure
Visually-Enhanced Food Behavior Checklist
Do you use this food label when shopping?
Plan, Shop, Save, and Cook Survey
How often do you shop with a grocery list?
How often do you compare unit prices before buying food?
Cooking Matters for Adults Survey
How often do you compare prices before you buy food?
How often do you use the “nutrition facts” on food labels? 
Feb 2015

16. Example
Feb 2015

17. Measuring Policy, Systems, and Environmental Changes
Feb 2015

18. RE-AIM Reach, effectiveness, adoption, implementation, maintenance
Reach, adoption, effectiveness, implementation, maintenance
Feb 2015

19. REACH (MT4-5)
Reach: Number of SNAP-Ed eligibles that benefitted from the change(s) during the period assessed (e.g., number of persons < 185% of Federal Poverty Level reached by the change in SNAP-Ed eligible settings).
Delineates at the environmental level where services are provided.
Short-term
Opportunities, partnerships, community champions
Medium-term
Description of changes adopted
Long-term
Is SNAP-Ed working towards a comprehensive approach?
WRO does not expect all the be covered, but at least two.
Feb 2015

20. ADOPTION (MT4-5)
Adoption: Aggregate number of SNAP-Ed settings where at least one organizational or environmental change is made in writing or practice to improve or strengthen access or appeal for healthy eating or physical activity during the period assessed
Feb 2015

21. Interpreting Reach and Adoption
MT4: Nutrition Supports Adopted 
Logic Model Component
Medium-Term Outcome –
Reach and adoption of nutrition policies or environmental changes
What to measure:
The documentation of change(s) adopted in the low-income setting and associated reach
Numerator:
Number of settings where at least one change is made in writing or practice to expand access or improve appeal for healthy eating
Denominator:
Number of settings in ST4, by category
How to Measure
ADOPTION
Documentation (direct observation, photograph) or interviews with key informants to confirm the uptake of the policy or environmental change in the low-income setting.
REACH
Means-tested setting
The number of SNAP-Ed eligibles, based on qualification for free or reduced price meals or federal poverty level, who benefit from the change(s).
For example, if the school cafeteria adopts changes in meal service, and there are 750 students who qualify for free or reduced price meals, the reach is 750.
Census tract, or census designated place
The number of SNAP-Ed eligibles, based on census data and direct observation, who benefit from the change(s). For example, if there are 6,000 (out of 10,000) individuals within 185 percent of FPL in a given census tract where a new farmers market opens, and on average in the past month, there have been 350 customers at the market per shopping day, multiply 350 x .60, which equals 210.
Feb 2015

22. Example
Feb 2015

23. IMPLEMENTATION (LT9)
Implementation: Aggregate number of SNAP-Ed settings that report a multi-component initiative with one or more organizational or environmental changes adopted AND at least one of the following: 1) evidence-based education,
2) marketing,
3) parent/community involvement,
4) staff training on continuous program and policy implementation.
Feb 2015

24. EFFECTIVENESS (LT9)
Effectiveness: Number of settings with improved food or physical activity environment assessment scores using a reliable and consistent environmental assessment tool.
Feb 2015

25. MAINTENANCE (I3)
Maintenance: Average percentage increase, or number of institutional or community resources invested in nutrition and physical activity supports or standards at SNAP-Ed settings in terms of staff (number of full time equivalents), cash, or in-kind supports
Feb 2015

26. Channels: Eat, Live, Learn, Work, Play, Shop
Eat: Restaurants, mobile vending/food trucks, congregate meal sites (or, other places where people primarily go to “eat”)
Live: Public housing, shelters, places of worship, community organizations, residential treatment centers, adult or senior services (or, other community or neighborhood settings where people “live” or live nearby)
Learn: Child care, head start, early care and education, adult education, schools, after-school, Cooperative Extension offices (or, other places where people go to “learn”)
Work: Worksites with low-wage workers5, job training programs, TANF worksites (or, other places where people go to “work”)
Play: Parks and recreation, YMCA, county fairs, Boys and Girls clubs, bicycling and walking paths (or, other places where people go to “play”)
Shop: Farmers markets, grocery stores, food retailers, food pantries (or other places where people “shop” for food)
Feb 2015

27. Feb 2015 Number of Sites by Category of Channel Eat Live Learn Work
NEOPB PSE Strategy
Eat
Live
Learn
Work
Play
Shop
Early Childcare
55 schools/ childcare centers
School Wellness Policies
29 schools
23 district offices
School Wellness Policy- Water Stations
1 city
23 schools
Farm to School
9 district offices
Joint Use Agreements
1 school
Healthy Retail
70 stores
Restaurant and Mobile Vending
2 vendors
Structured Physical Activity
1 public housing
5 churches
2 schools
Community/School Gardens
1 rehab center
4 apt complexes
18 churches
12 community sites
4 community centers
25 schools
1 park
Worksite
52 worksites
Safe Routes to Schools
2 neighborhoods
1 community center
14 schools
3 parks/
public land
Farmers' Markets
16 markets
4 produce stands
1 business facility
1 public land
Healthy Food and Beverage Standards
5 emergency food
28 churches
2 community centers
4 community-based organizations
3 WIC programs
2 cities
1 store
Feb 2015

28. Next Steps Selecting priority indicators for FY 16
Developing instructions and reporting book for priority indicators
Developing interpretive guide for 51 indicators
Ongoing collaboration with ASNNA Eval Sub-committee, Regional Centers of Excellence, NIFA, CDC, & HRSA for alignment of indicators
Feb 2015

29. Evaluation References
CDC. State indicator report on fruits and vegetables.
Report‐Fruits‐Vegetables‐2013.pdf
CDC. State indicator report on physical activity.
CDC. Recommended community strategies and measurements to prevent obesity in the United States. (COCOMO)
Leeman et al. An evaluation framework for obesity prevention policy interventions. Prev Chronic Dis 2012;9: Available at:
NIFA. Community Nutrition Education Logic Model.
Center for Training and Research Translation Web site. Available at:
Altarum Institute and RTI International. Cates, S., Blitstein, J., Hersey, J., Kosa, K., Flicker, L., Morgan, K., and Bell, L. Addressing the Challenges of Conducting Effective Supplemental Nutrition Assistance Program Education (SNAP-Ed) Evaluations: A Step-by-Step Guide. Prepared by Altarum Institute and RTI International for the U.S. Department of Agriculture, Food and Nutrition Service, March Available at:
RE-AIM Framework.
Healthy People
Feb 2015

30. Thank you!
(415)
Feb 2015