1. Plan, Evaluate and Improve Your Health Literacy Interventions and Activities Sabrina Kurtz-Rossi, M.Ed. Health Literacy Consultant April 10, 2013

2. Objectives After attending the session, participants will be able to: – Conceptualize a health literacy program evaluation – Write measurable health literacy outcome objectives – Identify indicators to assess both process and outcomes – Identify user-friendly evaluation resources for future use Olney, C. and Barnes, Susan (2006). Planning and Evaluating Health Information Outreach. NN/LM Outreach Evaluation Resource Center. Seattle, WA.

3. Ten Attributes of Health Literate Health Care Organizations – Brach C, Keller D, Hernandez LM, et al. Ten Attributes of Health Literate Health Care Organizations. Washington DC: National Academy of Sciences. 2012

4. Case Study Providence, RI Dorcas Place Adult and Family Learning Center Computer Lab Western Maine Auburn Public Library Computer Training Class

5. Research vs. Evaluation Research: To draw causal links between observed phenomena and to add to the knowledge base of the field Evaluation: To demonstrate effectiveness, and determine what works and what does not for the purpose of program improvement Boulmetis, J and Dutwin, P (2005). The ABCs of Evaluation. Jossey-Bass. San Francisco, CA

6. Health Literacy Program / Evaluation Planning Step 1: Know Your Audience Step 2: Identify Goals and Objectives Step 3: Develop Program (and evaluation) Plan Step 4: Implement and Evaluate Step 5: Apply and Share Results

7. Step 1: Know Your Audience Start with an Assessment – Identify needs in context – Guide your program – Build on already existing capacity – Identify champions within your organization Methods – Informal – discussion, observation – Formal – Key informant interviews, focused group discussions, survey

8. Attribute 1: In a Health Literate Organization leadership makes health literacy integral to its mission, structure, and operations. –

9. Case Study Methodology: Leaders at community-based organizations interviewed. Semi- structured interview protocol designed to explore community confidence in using the Internet to find health information and local resources to support access. All interviews conducted in person. Data were coded and analyzed into themes. Key Findings: Maine has a growing senior population and community-based organizations support public health efforts. Public schools, adult education, and libraries offer computer access and support (n=9). Latinos get their health information from sources they know and trust, including the radio. A network of minority health programs offer computers access, training and support (n=10).

10. Step 2: Identify Goals & Objectives Goal -- Why you are doing what you are doing and for whom Example: – Increase use of health literacy interventions – Improve community access to reliable online health information

11. Attribute 2: A Health Literate Organization integrates health literacy into planning, evaluation, patient safety, and quality improvement. –

12. Case Study Focused Health Information Outreach Project Goal: Increase access to health information among communities experiencing health disparities National Network of Libraries of Medicine – New England Region (NN/LM NER) http://nnlm.gov/ner/http://nnlm.gov/ner/

13. Step 3: Develop Program(and Evaluation) Plan Logic Model: A planning tool connecting activities to outcomes If you have these resources ResourcesActivities/OutputsOutcomes and conduct these activities you will get these outcomes

14. Why a Logic Model? Provides a framework for planning Helps organize your thoughts Clarifies intended outcomes Links activities to outcomes Acts as a communication tool ActivitiesResources OutputsOutcomes

15. Attribute 3: A Health Literate Organization prepares its workforce.

16. Case Study Goal: Increase access to health information among communities experiencing health disparities ResourcesActivitiesOutputsOutcomes Increase knowledge of NLM resources Increase use of NLM resources Increase community experts to help others Number of trainings held Number trained Types of materials developed Number of materials distributed Develop training materials Distribute materials at community events Consultant Outreach coordinators NLM materials NLM funding Conduct trainings Working student Librarians in communities

17. Activity Choose a goal (or use your own) – To increase staff awareness and use of health literacy interventions – To improve community access to reliable online health information – To improve health information literacy skills among patients and their families Complete the logic model worksheet - identify outcomes, activities, outputs, and resources

18. Attribute 4: A Health Literate Organization involves the population served in evaluation of health information and services.

19. Case Study Community Assessment – Key informant interviews Program – Tailored efforts based on findings – Community-based partnerships Evaluation – Outcome assessment: pre-post training evaluations, follow-up assessment – Process assessment: story-based evaluation, community partner exit interviews Western Maine (rural) Providence, RI (Latino)

20. Evaluation Plan Process – To determine what worked and what didn’t – For program improvement Outcome – To demonstrate effectiveness, value, and influence – For comparison purposes, to determine which programs are more effective

21. Attribute 5: A Health Literate Organization meets the needs of the populations and avoids stigmatization.

22. Process Assessment Methods – Activity logs – Teacher feedback forms – Focused group discussion – Interviews

23. Outcome Assessment Methods – Existing records – Pre/post tests – Follow-up surveys – Observation – Interview – Story-based

24. Attribute 6: In a Health Literate Organization providers and staff use health literacy strategies in verbal communication and confirm understanding.

25. Case Study Outcomes (what you want)Indicators (how you will know) Increased knowledge of NLM resources Increased use of NLM resources Increased community experts to help others Will report use two weeks later of follow up Will name resources on post- surveys Will document helping others on story-based evaluation forms Outcomes & Indicators

26. Measurable Objectives Four components of a measurable objective 1. Condition Following the training… After watching the video… 2. Who …the learner… …the patient… …the caregiver

27. Measurable Objectives 3. Performance (knowledge, attitudes, behavior) …will name… … will agree… … will use… 4. Criterion (quality/quantity) …5 out of 6… … at least 2… … the correct way…

28. Characteristics of a Measurable Objective Is audience focused, not intervention focused (the participant will…) Includes action verbs Targets one expectation Includes observable criteria for evaluation Matches goals to activities/strategies

29. Attribute 7: In a Health Literate Organization consumers have easy access to health information and services and navigation assistance.

30. Bloom’s Taxonomy Cognitive – explain, recall, list, predict, select, describe, define Affective – agree, avoid, help, join Psychomotor – use, develop, design, repair, taste, use, move

31. Verbs to Avoid Know Comprehend Understand Appreciate Study Gain knowledge of Learn

32. Attribute 8: A Health Literate Organization develops plain language materials that are easy to understand and act on.

33. Case Study ConditionWhoPerformanceCriterion Immediately after training participantswill name Two weeks after training participantswill report using Three months training participants NLM resources on post-survey NLM resources on follow up interview will write a story Describing how they helped others find information on story-based evaluation form Writing Measurable Objectives

34. Activity Choose an outcome (or use your own) – Increase confidence in using plain language/teach-back – Increased awareness of available online health information resources – Increased interaction with diabetes educators Complete the writing measurable objectives worksheet - identify condition, audience, performance, and criterion

35. Step 4: Implement and Evaluate Expect the Unexpected – Be flexible – Adjust to the environment – Address learners’ needs – Tweak your notion of success

36. Attribute 9: A Health Literate Organization addresses health literacy in high-risk situations, including transitions and about medicines.

37. Case Example Rhode Island: Pre/Post: Confidence in Ability to Find Health Information on the Internet Pre N= 52 0 Post N=503 Maine: Pre/Post: Ever Used NLM Resources Pre N=72 Post N=55

38. Case Example Successes – Training participants shared resources with friends and neighbors – Partners will continue to promote and distribute resources – Providers used resources to find accurate information to use in their work Challenges – Pre- and post-evaluations were hard to administer

39. Step 5: Apply and Share Results Apply Lessons Learned – Make changes based on what you learned – Replicate with similar or different audiences Share Your Findings – Partners – Colleagues

40. Attribute 10: At a Health Literate Organization they clearly communicates about health plans and health care costs.

41. Evaluation Resources Planning and Evaluating Health Information Outreach Projects Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach http://nnlm.gov/evaluation/guide/index.html

42. Questions & Comments Sabrina Kurtz-Rossi, M.Ed. Principal | Health Literacy Consultant 781-835-6488 sabrina@kurtz-rossi.com Kurtz-Rossi & Associates Health*Literacy*Education*Evaluation www.kurtz-rossi.com Member, Clear Language Group www.clearlanguagegroup.com Health Literacy Leadership Institute, June 10-14, 2013 http://healthliteracyleadreship.com