Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluating Partnerships: Why, When and How February 17, 2015 Presented in Collaboration with the Centers for Disease Control and Prevention B. J. Tatro,

Similar presentations


Presentation on theme: "Evaluating Partnerships: Why, When and How February 17, 2015 Presented in Collaboration with the Centers for Disease Control and Prevention B. J. Tatro,"— Presentation transcript:

1 Evaluating Partnerships: Why, When and How February 17, 2015 Presented in Collaboration with the Centers for Disease Control and Prevention B. J. Tatro, PhD ASTDD, 1

2 General Reminders This webinar will be recorded and archived on the ASTDD website (including speaker notes) Questions will be addressed after the speakers are finished. Please type your question into the “chatbox” that will appear at the end of the webinar and then click on the bubble to the right of where you type your question to send it to the moderator Please respond to the polling questions at the conclusion of the webinar 2

3 This presentation was supported by Cooperative Agreement 5U58DP from CDC, Division of Oral Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC 3

4 Desired Results State Oral Health Programs (SOHPs) have increased awareness of the importance of evaluating their partnerships and of engaging their partners in the evaluation process SOHPs have increased awareness of the importance of matching their partnership evaluation questions and tools to their partnership’s stage of development SOHPs are familiar with resources available to them to inform their planning, implementation, and use of partnership evaluation 4

5 Some Preliminary Thoughts on Partnerships What are they? – Nature of the partnership – Purpose of the partnership 5

6 Some Preliminary Thoughts on Partnerships Why do we need them? What does it take to develop, maintain, and grow a successful partnership? 6

7 Some Preliminary Thoughts on Evaluating Partnerships Why evaluate partnerships? How do we know if we are ready to evaluate a partnership? How do we approach the design and implementation of a partnership evaluation? 7

8 Step 1: Engage Stakeholders Identify who should be involved in the evaluation Specify the roles they will play Think about how to engage these stakeholders throughout the evaluation process 8

9 Step 2: Describe the Partnership Develop your logic model 9

10 10 State of Hawaii Purpose = Develop & coordinate partnerships to improve oral health Inputs  Activities  Outputs  Outcomes Hawaii Oral Health Team (staff) Internal Partners Family Health Services Division Programs:  Office of Primary Care & Rural Health  CSHCN  MCH (including MIECHV, Parenting Support, Adolescent & Youth, etc.)  WIC Other DOH:  Chronic Disease Management & Control Branch (Asthma, Cancer, Diabetes, Heart/Stroke, Obesity, Tobacco)  Office of Planning, Policy & Program Development (Preventive Health & Health Services Block Grant)  Developmental Disabilities External Partners State Medicaid Office Hawaii Federally Qualified Health Centers (FQHC) & Rural Health Clinics (RHC) Hawaii Chapter of the American Academy of Pediatrics Insurance providers Legislators Community representatives CDC Funding & TA ASTDD TA 1. Identify priorities & establish, enhance, & maintain a collaborative relationship with internal DOH partners, with priority focus on CDC-funded chronic disease programs, CSHCN, & MCH 2. Identify priorities & establish, enhance, & maintain a collaborative relationship with external partners, with priority focus on FQHCs & RHCs For both internal & external partners: a. Identify contacts b. Conduct 1:1 outreach c. Become familiar with each other’s mission & activities d. Identify shared interests & common goals e. Identify potential benefits of working together f. Identify options for collaborative efforts g. Make decision about whether & how to collaborate h. Establish partnership objectives & develop work plans i. Implement work plans j. Evaluate collaborative strategies a. Contacts identified, documented b. Communication established, documented c. Information shared, documented d. Shared interests & common goals identified, documented e. Benefits identified, documented f. Options identified, documented g. Decision made, documented h. Objectives established & work plans developed i. Work plans implemented, documented j. Evaluation conducted, documented Short Term Ongoing communication & collaborative action Increased awareness of the relevance of oral health to partners’ target populations Increased awareness of the benefits of prevention, early identification, & treatment to overall health  Intermediate Term Oral health addressed in partners’ burden documents if applicable, activities, messaging, plans, policies, &/or practices Resources devoted to oral health expanded/ leveraged  Longer Term Increased number & commitment of oral health champions Increased attention to prevention, early identification, & treatment Increased number of children have dental sealants  Impact Reduced dental caries, oral cancer, periodontal disease Improved health equity Improved overall health

11 Step 2: Describe the Partnership (continued) Identify what you need and what you have 11

12 12 Partnership Membership Assessment Tool from Fundamentals of Evaluation Partnerships/Evaluation Guide (National Health Disease & Stroke Prevention Program) WantHaveRolesWantHaveSkills/ExpertiseWantHaveRepresentation Partnership roles Data analysis, worksites State Emergency Services Leader Data analysis, healthcare State Obesity Program Committee leader Reviewer, medical content State Diabetes Program Task leader Writer State Tobacco Program Meeting planner Advocate for stroke State Epidemiology Meeting facilitator Advocate for heart disease State Office of Minority Health Strategic planner Legislative advocate Hospital Association Communications Medical expert Primary Care Association Training Cardiologist State Legislature/Policy Makers Financial support Neurologist Schools (as worksites) Content reviewer Healthcare quality improvement Community health clinics Budget management Nursing Private insurers Spokesperson Pharmacy Medicaid/Medicare Funder Media communications Prevention Research Center Champion, healthcare Workplace wellness Chambers of Commerce Champion, public health State policy change Unions Champion, worksites Community policy change Business coalition on health Strategic implementer Training for healthcare State American Heart Association Resource linker – connection to groups with influence or resources Evaluation Disparate groups (race/ethnicity, geographic, gender. SES. Etc.) Marketing

13 Step 2: Describe the Partnership (continued) Identify the stage of development of your partnership – Formation – Building – Maintenance Engage stakeholders 13

14 Step 3: Focus the Evaluation Design Develop and prioritize evaluation questions that are matched with your partnership’s stage of development Engage stakeholders 14

15 Step 4: Gather Credible Evidence Complete your evaluation plan: – Indicators – Data Sources (tools) – Methods of data collection – Frequency of data collection – Study design – Implementation roles, responsibilities, and timelines Engage stakeholders 15

16 16 State of Minnesota Evaluation Question Indicator/ Performance Measure MethodData SourcesFrequencyResponsibility How were the opportunities identified and leveraged to integrate oral health efforts with other programs that address related chronic disease/risk factors? # of meetings, campaigns, summits organized Meeting notes, summit reports review Meeting notes and summit reports Annually Program coordinator/ evaluator How many partnerships were developed with a focus on prevention interventions in a year? # of partnerships that focus on prevention intervention and that provided a letter of support or MOA Program reports review Letters of support Program reports Partnership database Annually Program coordinator/ evaluator How was the integration of oral health into overall health encouraged/institutionalized? # of meetings with other chronic disease partners Meeting notes and reports review Meeting notes and reports Biannually Program coordinator/ evaluator How was the documentation provided for collaboration with partners outside state health department? # of meetings, meeting notes, s, letters indicating agreement to collaborate Meeting notes and reports review Meeting notes and reports Biannually Program coordinator/ evaluator What are types of activities are partners involved in? # of specific tasks accomplished Meeting notes and reports review Meeting notes and reports Biannually Program coordinator/ evaluator What are the reasons for long term sustained partnerships? List of reasons for long term sustained partnerships Partners’ surveySurvey questionnaire Once in a grant period Program evaluator What are the challenges partners feel that hinder intended progress? List of reasons for long term sustained partnerships Partners’ surveySurvey questionnaire Once in a grant period Program evaluator

17 17 Evaluation Question Indicator/ Performance Measure MethodData SourcesFrequencyResponsibility How does the coalition focus on the following? Oral health infrastructure development and sustainability State oral health plan revision and implementation School-based/-linked sealant programs Community water fluoridation efforts Coalition initiatives in the areas of Oral health infrastructure development and sustainability State oral health plan revision School-based/-linked sealant programs Community water fluoridation Meeting notes and reports review Meeting notes and reports Biannually Program coordinator/ evaluator What sectors of CDC coalition diversity framework do coalition members represent? # of members from each sector of CDC coalition framework # of members from each sector attending coalition meetings Coalition member roster Member group is identified on the membership application Coalition member roster Stakeholder database On-going Program Coordinator Data Coordinator What were the steps taken to facilitate institutionalization and sustainability of coalition? 501c3 status, written vision/mission statement, priorities /plans/strategies Consistency of membership Review bylaws and coalition work plan By-laws and work planAnnually Coalition members/ program coordinator Which programs and policy changes have resulted from the coalition efforts? # of policy changes as a result of coalition efforts List of program initiated Review policy documents Review reports Policy document Coalition reports Annually Annually Coalition members/ program coordinator

18 18 Evaluation Question Indicator/ Performance Measure MethodData SourcesFrequencyResponsibility What types of communication are being used by partners? Types of communications by partners # of member that feel the communication is adequate Coalition website updated quarterly Review reports Member survey and review of agenda, minutes and reports Use of website for communications Progress reports Member survey and Coalition agendas, minutes and reports Coalition website On going On going Coalition secretary/ program coordinator How do collaborative partners focus on oral health issues and prevention strategies as outlined in the state oral health plan? # of partner involved in initiating new activities outlined in state oral health plan # of partner participating in activities outlined in state oral health plan Review program progress reports Program progress reports Quarterly Program coordinator/ evaluator What are the forms of documentation of commitment or support available from stakeholders e.g. delineating accountability and clear lines of responsibility for activities? Letters of support from stakeholders Description of roles and responsibilities of partners for activities Review coalition reports Review of coalition manual Coalition reports Coalition manual Annually Program coordinator How did partnerships influence oral health policies, practices and systems? List of policies, practices and systems influence Review report and survey results Annual reports and partners’ survey Once in the grant period Program evaluator

19 Step 5: Justify Conclusions Ensure accuracy of data and data entry Analyze the data Make sense of what you have learned Develop recommendations Engage stakeholders 19

20 Step 6: Ensure Use and Share Lessons Learned Develop a communication plan Communicate results Incorporate into ongoing planning Engage stakeholders 20

21 Resources ASTDD website (slides/speaker notes from this webinar will also be posted on the ASTDD website) and B. J. Tatro (ASTDD evaluation consultant) There are also a variety of resources related to oral health collaborations on the site. Collaboration: What Makes It Work (2 nd ed), P. Mattessich, M. Murray-Close, & B. Monsey (Fieldstone Alliance, 2001) Includes The Wilder Collaboration Factors Inventory. This publication includes a review of the literature on factors influencing successful collaboration and The Wilder Collaboration Factors Inventory TOOL. Collaboration: What Makes It Work (2 nd ed) Collaborative Effectiveness Assessment Activity (Prevention Institute, 2004) This TOOL is designed to stimulate thinking about some elements of effective collaboration. Collaborative Effectiveness Assessment Activity Evaluating Partnerships to Prevent and Manage Chronic Disease, F. Butterfoss, Preventing Chronic Disease, April 2009, Vol. 6, No. 2 This article describes how to develop a comprehensive evaluation strategy based on partnership theory, select outcome indicators, choose methods and tools, and use results for accountability and improving effectiveness. Evaluating Partnerships to Prevent and Manage Chronic Disease 21

22 Resources Evaluation Guide: Fundamentals of Evaluating Partnerships, National Heart Disease & Stroke Prevention Program (Centers for Disease Control and Prevention, 2008) This is one of a series of evaluation technical assistance tools. The guides supplement the CDC program evaluation framework. This particular guide provides information on how to implement the framework when evaluating partnerships at various levels (annual assessment, basic evaluation, and enhanced evaluation). Appendices include examples of evaluation questions, a variety of TOOLS, and other helpful information. Evaluation Guide: Fundamentals of Evaluating Partnerships Framework for Evaluation (CDC, 2012) CDC provides a six-step framework for conducting program evaluation. CDC also offers guides for developing logic models and developing evaluation plans; see Developing and Using a Logic Model and Developing an Effective Evaluation Plan Framework for EvaluationDeveloping and Using a Logic Model Developing an Effective Evaluation Plan Guide to Evaluating Collective Impact, H. Preskill, M. Parkhurst, & J. Splansky Juster (FSG, Collective Impact Forum, 2014) In addition to three guides, this site links the user to a knowledge exchange related to collective impact. Guide to Evaluating Collective Impact 22

23 Resources Handbook on Planning, Evaluating and Improving Interagency Collaboration in State Oral Health Programs (ASTDD, 2012) The handbook and companion workbook provide a framework for evaluating collaboration between state oral health programs and other entities. A variety of TOOLS, as well as oral health examples, are provided. Handbook on Planning, Evaluating and Improving Interagency Collaboration in State Oral Health Programs Measuring Collaboration Among Grant Partners, B. Frey, J. Lohmeier, S. Lee, & N. Tollefson, American Journal of Evaluation, 2006, 27, 383 This TOOL can be used to assess levels of collaboration—ranging from networking to collaboration for each partner. Measuring Collaboration Among Grant Partners PARTNER This is an online social network analysis tool designed to collect, analyze, and interpret data to improve collaboration within community networks. PARTNER 23

24 Resources Partnership Evaluation: Guidebook and Resources, National Center for Chronic Disease Prevention & Health Promotion, Division of Nutrition, Physical Activity & Obesity (Centers for Disease Control and Prevention, 2011) This is one of a series of evaluation technical assistance tools. The guides supplement the CDC program evaluation framework. This particular guide provides information on how to implement the framework when evaluating partnerships. Appendices include a variety of TOOLS and other helpful information. Partnership Evaluation: Guidebook and Resources Partnership Self-Assessment Tool (Center for the Advancement of Collaborative Strategies in Health, 2002) This TOOL can be used to assess how well a collaborative process is working and to identify specific areas for improvement. It is designed for partnerships of five or more that have been in existence for at least six months. Partnership Self-Assessment Tool Tools for Partnership Evaluation (CDC, 2012) This “coffee break” webinar was presented by S. Ladd, CDC Division for Heart Disease and Stroke Prevention Applied Research and Evaluation Branch. It covers three types of TOOLS— evaluability assessment, membership and roles, and process and functioning. Tools for Partnership Evaluation 24

25 Thank You! Thank you to all who provided information for and input into this webinar, including: – Harder+Company, for sharing a matrix of partnership/collaboration evaluation tools that they researched and prepared – John Shoemaker, for sharing a variety of partnership/collaboration tools and other resources – The many CDC-funded states that provided copies of their partnership/collaboration evaluation logic models and evaluation plans, especially those who allowed us to draw examples from their work And thanks to all of you who participated today! 25

26 Polling Questions Are you planning to do a partnership evaluation within the next few years? – Yes, No, Not Sure Have you developed your partnership evaluation plan yet? – Yes, No, Not Applicable (no study planned) What could ASTDD do to help you plan/conduct your partnership evaluation? 26


Download ppt "Evaluating Partnerships: Why, When and How February 17, 2015 Presented in Collaboration with the Centers for Disease Control and Prevention B. J. Tatro,"

Similar presentations


Ads by Google