1. National Network of Libraries of Medicine Outreach Evaluation Resource Center Planning Outcomes-Based Outreach Programs

2. You got to be very careful if you don’t know where you’re going because you might not get there -- Yogi Berra

3. Why talk about outcomes in the planning phase? What gets measured gets done If you can’t recognize failure, you can’t correct it If you cannot see success, you can’t reward it (and may be rewarding failure) If you do not measure results, you cannot see success If you can demonstrate results, you can win public support From Reinventing Government

5. A plan helps you move in the right direction. Evaluation keeps your eyes on the road

6. By the end of this workshop you will know how to Use logic models to plan programs around desired results (outcomes) Shape objectives from outcomes Develop evaluation plans based on logic models

7. The Logic Model Creating the map

8. Begin with the results you’re seeking

9. Figure out what you have to do

10. Then identify what you need

11. Activities are more under your control than outcomes

12. Here are examples of the items for each column

13. Goal: Have a good summer vacation

16. Vacation: Output versus Outcome

17. Library Project: Output Focus

18. Library Project: Outcome Focus

19. There are two general categories of outcomes

20. Examples of system-level change

21. People change in stages

22. Kirkpatrick’s Model of Program Evaluation is popular for evaluating training programs

23. Kirkpatrick’s Model can be useful in completing the Logic Model

24. Last step: Get Feedback from stakeholders and key informants

25. How could the logic model help plan proposals? Logic ModelProposal InputsBudget ActivitiesStrategies OutputsDeliverables OutcomesResults & Evaluation InfluencesSupport & Barriers AssumptionsReviewers’ questions

26. Use the logic model to write final reports Logic ModelReport InputsResources you acquired/used ActivitiesWhat you did OutputsDeliverables Outcomes What you accomplished Influences Lessons learned Assumptions Lessons learned

27. Making outcomes measurable: Writing objectives

28. Build objectives around your outcomes

29. Indicators are the observable signs of the outcome Outcome Parents will increase their ability to find online health information Indicator (measurable) Parents will report they were able to use MedlinePlus to locate information on at least one health topic that was important to them

30. The target sets the criteria for success Indicator (measurable) Parents will report being able to use MedlinePlus to locate information on at least one health topic that was important to them Target (criterion for success) 70% of participants

31. Time frame tells you when you are going to check on the indicator

32. Now you can write the outcome objective for an outcome Outcome Parents will increase their ability to find online health information Outcome Objective By the end of the training session, 70% of parents will report being able to use MedlinePlus to locate information on at least one health topic that was important to them

33. Example Objective: Awareness Outcome The OERC will improve access to information about emerging evaluation trends among NN/LM members Outcome Objective By the May 2015, All 8 regional medical libraries will have re-posted OERC blog articles in their Newsletters or blog

34. Objective Example: Impact Outcome Clinicians will recommend health information to their patients Outcome Objective One month after training, 60% of clinicians will report providing at least one patient with Information about MedlinePlus

35. Discussion: What indicators could you use? Increased use of health resources Indicator: ____________________ More access to health resources Indicator: ____________________ More cost effectiveness Indicator: ____________________ Increased efficiency Indicator: ____________________

36. Webinar 4: Adding Evaluation to Your Plan

37. What your stakeholders want to know  Did you get there?  Did you do everything you planned?  How Did you get there?  Did you do everything you planned?  What unexpected things happened?  Would you go there again?  What advice do you have for others?  What would you do differently next time?  What nexpected things happened?  Would you go there again? What advice do you have for others? What would you do differently next time?

38. Focus on “Rapid” methods Short surveys Site visits Observations Key informant and focus group interviews Existing data review

39. Use mixed methods Quantitative methods for numbers Qualitative methods for text or images

40. Source and Method Where we find dataHow we collect data

41. Process evaluation: Keeping your eyes on the road

42. Process assessment is developed for planned activities

43. Documenting process

44. How to describe progress

45. Add check-ins with key players

46. Comments and feedback allow you to assess quality of your program

47. Outcomes Assessment: Did you make it and was it worth it?

48. Outcomes assessment is guided by the “intended results’ columns

49. How do you evaluate your outcome objective? By the end of the training session, 70% of parents will report being able to use MedlinePlus to locate information on at least one health topic that was important to them DATA SOURCE Participants EVALUATION METHOD Session Evaluation Form DATA COLLECTION TIMING At the end of each training session

50. Identify Source, Method, Timing By the May 2015, All 8 regional medical libraries will have re-posted OERC blog articles in their newsletters or blog DATA SOURCE NN/LM websites EVALUATION METHOD Review of blogs and newsletters DATA COLLECTION TIMING At the end of each quarter

51. Example: ¡VIVA! Outreach Project Objectives: When talking with patients, physicians and physicians assistants will increase use of : Plain language Open-ended questions Teach back Patient health information that matches language and literacy needs MedlinePlus as a patient information source

52. Example: The STISD Health Literacy Internship Project

55. Most Significant Change Method

56. Basic MSC Questions What is the most significant change you have experienced since________ Why was this change so significant to you? What different has it made already? What difference will it make in the future?

57. NN/LM Outreach Evaluation Resource Center Visit us here: Website: http://nnlm.gov/evaluation/ Blog: http://nnlm.gov/evaluation/blog Facebook: https://www.facebook.com/nnlmOERC Twitter: @nnlmOERChttp://nnlm.gov/evaluation/http://nnlm.gov/evaluation/blog: @nnlmOERC Cindy Olney Acting Assistant Director olneyc@uw.edu Karen Vargas Evaluation Specialist kjvargas@uw.edu