1. Starting With the End in Mind: Capturing Results Sarah Thach, NC DHHS Office of Healthy Carolinians & Health Education September 24, 2010

2. What words or feelings come to mind when you think of evaluation?

3. Session objectives Inspire you that evaluation can be fun, creative, illuminating, even joyous! Discuss purposes of evaluation Review core steps in evaluation Share resources and tools Define evidence-based interventions (EBIs), discuss why use then, where to find them, how to adapt them Practice telling your program’s story

4. Why evaluate?

5. Assess Evaluate Implement Plan

6. Why Evaluate? 1. Prove what you’re doing 2. Improve what you’re doing According to a recent study, one of the top predictors of nonprofit sustainability is spending time leveraging evaluation data for making meaning, decision making and planning, not primarily for accountability or validation (Peter York, TCC Group) With thanks to Jill Fromewick of Summit Research Associates

7. Key Steps in Evaluation 1. Start with the end in mind: what do you want to accomplish? Desired outcomes 2. Figure out how you can achieve it Logic model, Theory of change, Evidence-based interventions 3. Figure out how to measure it Evaluation Plan 4. Measure it and adjust program as needed 5. Tell your story

8. Step 1: Start With the End in Mind

9. Ultimately, what change do you want to see in the health status of the population you’re serving?

10. Step 2: Figure out how to achieve it

11. Creating a Road Map

12. 12 University of Wisconsin-Extension, Program Development and Evaluation Tool: Logic model What we do Who we reach Desired results INPUTS OUTPUTS OUTCOMES Resources: $, staff/ volunteers, eqpt, space ActivitiesParticipation Short (1-3 yrs) Know- ledge, Attitudes, Skills Medium (4-6 yrs) Behav- iors, social norms, environ ment Long- term 7-10 yrs Ultimate goal What we invest

13. Tool: A Theory of Change helps make connections between activities and desired outcomes

15. Use Evidence-Based Interventions Evidence-Based Interventions have been… Implemented Evaluated Found to be effective (at least in one particular setting and population) Evidence-Based Interventions can include… Strategies Pre-packaged “soup-to-nuts” programs With thanks to NCTraCS

16. A continuum of evidence: Weaker… …Stronger New intervention: - "Hey, I have an idea!" - "Sure, why not?" Theory- driven new intervention “Promising practice” Evidence- based: Intervention implemented, evaluated, and found to be effective “Best practice” Research tested: Intervention found effective in a control study (some sites had intervention, some didn't) Systematic review of multiple intervention trials shows effectiveness

17. Why Use EBIs? Provide a recipe or road map so you don’t have to reinvent the wheel Are likely to succeed Help you use scarce resources wisely: $, time, volunteers/partners Are increasingly required by funders

18. Tool: Finding & Using EBIs

19. Finding EBIs The Community Guide to Preventive Services Cochrane Reviews Canadian Best Practices Portal Databases of Journal Articles: PubMed search engine for medical articles Google Scholar (includes articles that are not peer-reviewed) Content-Specific Databases: AHRQ Health Care Innovations Exchange US Preventive Services Task Force Recommendations SAMHSA’s National Registry of Evidence-based Programs and Practices

20. NC databases of interventions CareShare Health Alliance’s Knowledge Bank https://www.caresharehealth.org/https://www.caresharehealth.org/ Healthy Carolinians: CHA priorities and action plan strategies http://www.healthycarolinians.org/assessmen t/healthobjectives.aspx http://www.healthycarolinians.org/assessmen t/healthobjectives.aspx NC Center for Public Health Quality’s Proven and Promising Practices http://www.ncpublichealthquality.org/ctr/ http://www.ncpublichealthquality.org/ctr/

21. Assessing EBIs for a good fit Think about: 1. Will it work for your target group’s age, culture, readiness to change? 2. Will it work for your host organization’s budget, staffing capabilities, timeframe? 3. Will it work in your community setting and synergize with existing resources/programs? Contact program authors for further info

22. Adapting EBIs for a good fit You can modify an EBI to make it more culturally- relevant to your target audience or community setting without compromising the intervention’s effectiveness as long as you stay true to the core of the program Include community partners and staff Stay true to the core program Record reasons for changes See National Implementation Research Network for tips on effective implementation http://www.fpg.unc.edu/~nirn/

23. Step 3: Figure out how to measure it

25. 25 University of Wisconsin-Extension, Program Development and Evaluation What do you measure? What we do Who we reach Desired results INPUTS OUTPUTS OUTCOMES Resources: $, staff/ volunteers, eqpt, space ActivitiesParticipation Short (1-3 yrs) Know- ledge, Attitudes, Skills Medium (4-6 yrs) Behav- iors, social norms, environ ment Long- term 7-10 yrs Ultimate goal What we invest Process EvaluationOutcome Evaluation

26. Tool: Evaluation plan Activities and outcomes Indicators: observable measures that describe how well outcomes have been achieved Data collection methods, sourceFrequency and schedule of data collection Using the logic model, identify activities & outcomes you want to measure For each activity/outcome, identify at least one indicator How will you collect data? Methods include interviews, focus groups, surveys, participant observation, document review, secondary data, analysis of existing databases From who/ what entity? How often does data collection take place? One time, twice, multiple times, continuously? When?

27. A compelling argument appeals to the… Head & WalletHeart

28. Make sure your evaluation includes… Data StoriesAnecdotesTestimonials Cost/Benefit AnalysisReturn on Investment$ Saved$ Generated

29. The hunt for good measures

31. You know it’s hot in NYC when… Barron J. (7/22/10) New York Times “No Matter How You Cut It, a July That’s Too Hot”

32. What about access to care? Outcomes: Decrease in ER use for primary care Increased appropriate use of primary care % uninsured with a medical home How long uninsured wait to get care after symptoms appear Collection: Self-reported vs. provider-reported vs. existing statistics

33. What about access to care? Process measures you care about: Collaboration: # partners # lead agencies for various elements of intervention # agencies contributing funding towards initiative/ inkind quantified # meetings attended Self-reported ownership of the network Participants can articulate what collaborative is and its goals Seeing collaborative as distinct from their own organization Cohesiveness (how was the collaborative named?)

34. Evaluation plan: access to care Activities and outcomes Indicators: observable measures that describe how well outcomes have been achieved Data collection methods, sourceFrequency and schedule of data collection Using the logic model, identify activities & outcomes you want to measure For each activity/outcome, identify at least one indicator How will you collect data? Methods include interviews, focus groups, surveys, participant observation, document review, secondary data, analysis of existing databases From who/ what entity? How often does data collection take place? One time, twice, multiple times, continuously? When?

35. Step 5: Measure it and Adjust Program as Needed

36. Step 6: Tell Your Story

37. Telling Your Story We are a storytelling species! “Facts don’t have the power to change someone’s story. Your goal is to introduce a new story that will let your facts in.” – Annette Simmons

38. Tool: Storytelling Tips Components of a good story: Start with a common assumption Introduce a point of conflict Cast the story with clear heroes and villains Include at least one memorable fact Point the way to a happy ending Source: Terrence McNally

39. “Sticky” (Memorable) Stories: Simple Unexpected Concrete Credible Emotional Stories Stick Source: Heath D & Heath C. 2007. Made to Stick: Why Some Ideas Survive and Others Die.

40. Core stories to collect 1. Nature of our challenge To remind staff/volunteers why their time and energy is needed everyday 2. How we began How the group was founded – captures both the need for the work and the specific approach you’ve taken 3. Emblematic successes Demonstrate group’s effectiveness over time 4. Individual Performance Shows professionalism, creativity, commitment your group brings to the challenge 5. Lessons learned Remind your group that occasional misfires are inevitable and should be embraced for what they can teach you

41. Sample story

42. In pairs, take turns telling a story about your program: Storytellers, select a story re.: Nature of our challenge How we began Emblematic successes Individual performance Lessons learned Listeners, listen for: Simple Unexpected Concrete Credible Emotional Stories Stick

43. Showing your story

44. “The best statistical graph ever” Minard’s map of Napoleon’s 1812 Russian campaign

46. From Swain Partnership for Health annual report

48. Resources Evaluation www.cdc.gov/eval/evalcbph.pdf http://www.cdc.gov/eval/resources.htm http://www.uwex.edu/ces/pdande/evaluation/pdf/tobaccomanual.pdf http://www.uwex.edu/ces/pdande/evaluation/index.html http://meera.snre.umich.edu/plan-an-evaluation/plonearticlemultipage.2007- 10-30.3630902539/participatory-evaluation Logic Models and Theories of Change http://www.hfrp.org/publications-resources/browse-our-publications/how-to- develop-a-logic-model-for-districtwide-family-engagement-strategies http://www.hfrp.org/publications-resources/browse-our-publications/how-to- develop-a-logic-model-for-districtwide-family-engagement-strategies http://www.aecf.org/upload/publicationfiles/cc2977k440.pdf http://www.publichealth.arizona.edu/chwtoolkit/pdfs/Logicmod/chapter3.pdf Telling Your Story http://discovery.wcgmf.org/resources/sps_resource_942.pdf Heath C & Heath D 2007. Made to Stick: Why Some Ideas Survive and Others Die.