1. PHAB's Approach to Internal and External Evaluation Jessica Kronstadt | Director of Research and Evaluation | November 18, 2014 APHA 2014 Annual Meeting

2. Improved community health indicators / reduced health disparities Organizational structure Board, committees and work groups Staffing and expertise Principles for standards, measures, and assessment process Site visitors Interest, buy-in and commitment to seek accreditation Appropriate stability, resources, and readiness to apply Previous quality improvement and assessment experience Increased visibility and credibility of public health agencies Ultimate Outcomes Improved responsiveness to community priorities Public Health Agency Accreditation System Approved December 2013 Enhanced internal and external collaboration Legend Accrediting Agency Individual Public Health Agencies Stakeholders and Partners Public Health Field Funders Partners at national, state, regional, and local levels Funding Incentives Technical Assistance Researchers and research networks Improved conditions in which people can be healthy Improved identification and use of evidence- based practices and policies Market program Implement the 7 steps of accreditation Train agencies and site visitors Develop e-PHAB Evaluate program and improve quality Promote research Promote national accreditation Encourage agencies to meet national standards and seek accreditation Support agencies through TA before, during, and after process Conduct and disseminate research Participate in training and TA Assess readiness Submit application and documentation Host site visit Review and share findings Develop and implement improvement plan Implement QI Mentor other agencies Participate in reaccreditation process Accreditation program: marketed, implemented, evaluated, and improved e-PHAB developed and data captured National consensus standards for public health agencies Communication efforts delivered Technical assistance, trainings, and QI tools provided Research conducted and disseminated Agencies are accredited Report received and acted on QI efforts are in place Agencies are mentored Plans for reaccreditation underway Increased science base for public health practice Increased support for accreditation Increased knowledge of organizational strengths and weaknesses Increased consistency in practice Increased use of benchmarks for evaluating performance Increased organizational accountability Increased capacity for optimal investment in public health Increased public recognition of public health role and value Intermediate Outcomes Proximate Outcomes OutputsStrategiesInputs Increased use of proven QI methods and tools resulting in improvements in practice Increased inter-agency and inter-sectoral collaboration Public health agencies more effectively and efficiently use resources Strengthened organizational capacity and workforce Strong, credible and sustainable accreditation program in place Increased awareness of importance of QI and a supportive culture Improved communication about public health Strengthened public health agencies and systems Standards adopted as performance measures Standards drive public health transformation

3. Overview Internal evaluation External evaluation Program data – Annual Reports

4. Internal Evaluation Overview Primarily focused on process – Allows PHAB to make informed decisions about improving the accreditation process Data Collection – Health Department Surveys (n = 63) – Site Visitor Surveys – Training evaluations – PHAB Accreditation Specialist Surveys

5. Health Department Experience

6. External Evaluation Overview Initial 3-year contract Focus on process and short-term outcomes Data collection from HDs – Survey 1: After HDs submit their Statement of Intent (n=122) – Survey 2: After HDs are accredited (n=28) – Survey 3: One year after HDs are accredited (n=17) – Interviews with 18 HD staff/stakeholders

7. Perceived Benefits Data reported by applicant HDs prior to participating in PHAB training – Accreditation will stimulate quality and performance improvement opportunities (100% Strongly Agree or Agree) – Accreditation will allow HD to better identify strengths and weaknesses (98% Strongly Agree or Agree) – Accreditation will improve management processes used by HD leadership team (98% Strongly Agree or Agree)

8. Benefits and Outcomes One Year Post Accreditation Survey of health departments accredited one year, n=17

9. Quality Improvement Quality Improvement Outcomes % Strongly Agreed or Agreed Shortly after Accredited (n=28) 1 Year After Accredited (n=17) Documentation selection and submission process helped identify areas for performance and quality improvement initiatives. 100%N/A Because of accreditation, we have implemented or plan to implement new strategies to monitor and evaluate effectiveness and quality. 100%N/A As a result of accreditation, we have implemented or plan to implement new strategies for quality improvement. 89%100% As a result of accreditation, we have used or plan to use information from our QI processes to inform decisions. 100%94% As a result of the accreditation process, our health department has a strong culture of quality improvement. N/A88%

10. “The way we document things has changed. Since we have been through the accreditation process, we know it is very important to have proof that you did do something. Our Board of Health has been supportive, and we give them updates on where we are in the process. [Accreditation] has helped our agency in a lot of ways.” Applicant health department, on early outcomes experienced prior to the PHAB Site Visit

11. Annual Reports Overview Required for accredited HDs Purpose – Continue communication with PHAB concerning conformity with standards and measures – Support health department in sustaining and advancing its quality improvement culture – Support health department in being prepared for reaccreditation Includes descriptions of: – Improvement activities – CHA, CHIP, strategic plan, QI plan

12. 14 Annual Reports describe 29 Improvement Activities –Getting into compliance with mandated frequencies of inspections –Improving a program that works with schools to implement environmental/policy changes –Improving communications with governing entity –Procuring an EMR system to get better data for evaluation & performance management –Improving new employee orientation –Streamlining & strengthening process for responding to grant RFPs Almost all indicated progress towards goals –“We also feel that [HD’s] PHAB accreditation status demonstrated our commitment and value to our community and policy makers.” IMPROVEMENT ACTIVITIES

13. 8 of 14 HDs reported improvements in health indicators in CHIP Examples of activities related to prerequisites and QI plan –Incorporating Essential Public Health Services and PHAB domains into all job descriptions and annual employee evaluations –Coordinating with a local hospital for their IRS requirements for the CHA –Engaging all divisions in the HD in at least one QI project in the past fiscal year –Providing data to community partners as part of efforts to address social determinants of health (e.g., high school graduation rates) –Expanding opportunities for community involvement in CHA and partner engagement in CHIP CONTINUING PROCESSES