1. Public Health Performance Measurement Moving from Version 1.0 to 2.0 AcademyHealth Annual Meeting Boston, MA June 26, 2005

2. University of Kentucky Team Members F. Douglas Scutchfield, MD, P.I. F. Douglas Scutchfield, MD, P.I. Evelyn A. Knight, PhD Evelyn A. Knight, PhD Ann V. Kelly, MHA Ann V. Kelly, MHA Ilie Puiu Vasilescu, PhD (UVa-Wise) Ilie Puiu Vasilescu, PhD (UVa-Wise) Michelyn Bhandari, MPH Michelyn Bhandari, MPH

3. Goals of the National Public Health Performance Standards Program (NPHPSP) Provide performance standards for public health systems Provide performance standards for public health systems Improve quality and accountability of public health practice Improve quality and accountability of public health practice Conduct systematic collection and analysis of performance data Conduct systematic collection and analysis of performance data Develop a science-base for public health practice improvement Develop a science-base for public health practice improvement

4. Aims of the Assessment Instruments Provide a NPHPSP performance report to the public health agency and the state or local system. Provide a NPHPSP performance report to the public health agency and the state or local system. Provide baseline, feedback and guidance to the public health system for continuous performance improvement. Provide baseline, feedback and guidance to the public health system for continuous performance improvement.

5. Four Concepts Applied in NPHPS 1. Based on the 10 Essential Public Health Services (EPHS) 2. Focus on the overall public health system 3. Describe an optimal level of performance 4. Support a process of quality improvement

6. Goals of UKYs Instrument Research (Local and State PHPA) Decrease the burden on users due to instrument characteristics. Decrease the burden on users due to instrument characteristics. Assure content supports the goals of the performance assessment process. Assure content supports the goals of the performance assessment process. Increase the usability of the results for performance improvement. Increase the usability of the results for performance improvement.

7. Research Phases Preliminary interviews--determine areas of need for instrument improvementstate and local instruments. Preliminary interviews--determine areas of need for instrument improvementstate and local instruments. Psychometric analysis--identify items that might be changed or eliminated in future versionslocal instrument only. Psychometric analysis--identify items that might be changed or eliminated in future versionslocal instrument only. Discussion groups--identify specific changes in question inclusion and/or wording which would improve instrument usability and usefulness state and local instruments. Discussion groups--identify specific changes in question inclusion and/or wording which would improve instrument usability and usefulness state and local instruments.

8. Preliminary InterviewsWho? State instrument : State instrument : –5 states –10 informants Local instrument: Local instrument: –6 state coordinators (12 informants) –Sample of local users by jurisdiction size, geographic region Stopped at five interviews--no new information Stopped at five interviews--no new information 3 states (5 informants) 3 states (5 informants)

9. Preliminary InterviewsHow? Semi-structured--questions about Semi-structured--questions about –Organization of PH in each state –Training & preparation for assessment –Assessment process –Instrument scoring –Instrument characteristicse.g. Clarity Clarity Completeness Completeness Lengthnumber of questions Lengthnumber of questions Usefulness Usefulness

10. Preliminary InterviewsResults Directions clear; some questions clear, some not Directions clear; some questions clear, some not Understandable by public health--but not partners Understandable by public health--but not partners Too long, instrument fatigue--but details important Too long, instrument fatigue--but details important Rating scale clearbut doesnt correspond with report Rating scale clearbut doesnt correspond with report Rating methodology variesconsensus, vote, etc. Rating methodology variesconsensus, vote, etc. Subjective scoresdepend on who is present Subjective scoresdepend on who is present Costly to gather information needed to answer the questions Costly to gather information needed to answer the questions

11. IndicatorLevel 2 Model Standard MeasureLevel 3 (First-tier stem) Essential ServiceLevel 1 Instrument Format MeasureLevel 4

12. MeasureLevel 5 Summary Questions-- Not analyzed MeasureLevel 4 (Second-tier stem)

13. Numbers of Indicators & Questions (Local Instrument) EPHS Indicators Level 3 stem Qs Level 4 Qs Level 5 Qs Total 1. Monitor Health Status 321512496 2. Diagnose & Investigate 41951979 3. Inform, Educate & Empower 27331555 4. Mobilize Community Partnerships 2738449 5. Develop Policies & Plans 412253673 6. Enforce Laws & Regulations 31126037 7. Link People to Health Services 3945054 8. Assure Competent Workforce 416411572 9. Evaluate 313382576 10. Research 31123034 TOTAL 31126371128 625

14. Performance ScoringLevels 3 & 2 Level 3 stem scores are weighted scores based on stem and level 4 & 5 sub-questions. Level 3 stem scores are weighted scores based on stem and level 4 & 5 sub-questions. Stem question (3-digit) Stem question (3-digit) Sub-questions (4-digit) Sub-sub-questions (5 digit) Level 2 scores (model standards) are average of Level 3 scores--yields 31 Indicator Scores. Level 2 scores (model standards) are average of Level 3 scores--yields 31 Indicator Scores.

15. Performance Scoring Level 1 & Overall Indicator Scores are averaged to obtain 10 EPHS scoresLevel 1. Indicator Scores are averaged to obtain 10 EPHS scoresLevel 1. 10 EPHS scores are averaged to obtain one overall Public Health Performance Score. 10 EPHS scores are averaged to obtain one overall Public Health Performance Score.

16. Psychometric Analysis of Local Instrument Determine Cronbachs internal reliability Determine Cronbachs internal reliability Determine Item-Total Correlations (ITC) Determine Item-Total Correlations (ITC) Test effect on reliability of removing specific questions or tiers of questions Test effect on reliability of removing specific questions or tiers of questions Use results to frame discussion group interviews Use results to frame discussion group interviews

17. EPHS Overall Chronbachs if EPHS Removed Item-total Correlation Coefficient 1. Monitor Health Status 0.908 0.660 2. Diagnose & Investigate 0.918* 0.448 3. Inform, Educate, Empower 0.906 0.697 4. Mobilize Partnerships 0.909 0.657 5. Develop Policies & Plans 0.897 0.840 6. Enforce Laws & Regulations 0.909 0.645 7. Link People to Health Services 0.905 0.703 8. Assure Competent Workforce 0.896 0.717 9. Evaluate 0.890 0.797 10 Research 0.896 0.719 *Reliability of group to total increases slightly if EPHS #2 (item) is removed. Reliability is slightly higher with only 9 EPHS (0.918), than with all 10 EPHS (0.915). For Overall Score = 0.915 (very reliable) Table 1. Internal Reliability of Overall Performance Score Based on 10 EPHS Scores (n=228)

18. EPHS # Indicators EPHS Chronbachs 1. Monitor Health Status 3 0.651 2. Diagnose & Investigate 4 0.753 3. Inform, Educate, Empower 2 0.832 4. Mobilize Partnerships 2 0.778 5. Develop Policies & Plans 4 0.748 6. Enforce Laws & Regulations 3 0.651 7. Link People to Hlth Services 3 0.620** 8. Assure Competent Workforce 4 0.695 9. Evaluate 3 0.847 10 Research 3 0.787 Table 2: Internal Reliability of 10 EPHS Scores based on Indicator Scores (n=228)

19. Summary of Internal Reliability Overall & EPHS Instrument has very high reliability overallmeasuring the PH construct Instrument has very high reliability overallmeasuring the PH construct EPHS-level reliability varies EPHS-level reliability varies –EPHS 3,9Very reliable –EPHS 2,4,5,10Acceptable reliability –EPHS 8Marginal reliability –EPHS 1,6,7Not reliable

20. Table 3. Internal ReliabilityIndicators based on Level 3 First-Tier Stem Question Scores EPHS Ind #1 Ind #2 Ind #3 Ind #4 1. Monitor Health Status** 0.9500.8020.593-- 2. Diagnose & Investigate 0.6890.7300.7390.766 3. Inform, Educate, Empower** 0.7670.687-- 4. Mobilize Partnerships 0.8050.825-- 5. Develop Policies & Plans 0.4970.7380.8180.826 6. Enforce Laws & Regulations** 0.5680.8250.796-- 7. Link People to Health Services** /10.8520.801-- 8. Assure Competent Workforce** 0.8770.7080.6930.886 9. Evaluate** 0.9190.8610.798-- 10 Research 0.7880.8390.790--

21. Summary of Internal Reliability of 31 Indicators 14 indicators very reliable >0.8 14 indicators very reliable >0.8 10 indicators of acceptable 10 indicators of acceptable reliability 0.7 to 0.8 –3 become very reliable if one stem question is removed 6 indicators not reliable <0.7 6 indicators not reliable <0.7

22. Psychometrics of Current and Simulated Instrument Versions (V.1-V.4) Current Local Assessment Instrument Current Local Assessment Instrument –V.1All questions as scored by CDC Eliminate scores of drill-down details Eliminate scores of drill-down details –V.2No level 5 detailsuse only level 3&4 –V.3No level 4&5 detailsuse only level 3 stem –V.4No questions with high or ns correlation to stems at levels 4&5

23. Table 4. Comparison of Reliability across the Four Versions (v.1-4) for 10 EPHS

24. Table 5. Internal Reliability10 EPHS Scores based on Indicator Scores V.1 and simulated V.2-V.4 EPHS V.1 V.2 V.3 V.4 1. Monitor Health Status 0.6510.6450.5530.649 2. Diagnose & Investigate 0.7530.7500.7320.754 3. Inform, Educate, Empower 0.8320.8300.6990.823 4. Mobilize Community Partnerships 0.7780.7690.7310.768 5. Develop Policies & Plans 0.7480.7460.7230.760 6. Enforce Laws & Regulations 0.651 0.6200.641 7. Link People to Health Services 0.620 0.5630.610 8. Assure Competent Workforce 0.6950.6900.6210.690 9. Evaluate 0.8470.8440.8280.835 10 Research 0.7360.7330.6820.732

25. Moving from V1.0 to V2.0 Instrument construction Instrument construction Instrument content Instrument content Instrument context Instrument context Instrument consensus Instrument consensus

26. Instrument Construction Clarity Clarity Time Time Complexity Complexity Subjectivity Subjectivity

27. Instrument Content Examine unreliable EPHS and subscales Examine unreliable EPHS and subscales Determine key content Determine key content Fit content into a logic model with inputsoutputs--outcomes Fit content into a logic model with inputsoutputs--outcomes

28. Instrument Context Large vs small systems Large vs small systems Centralized vs decentralized Centralized vs decentralized Urban vs rural Urban vs rural

29. Instrument Consensus Agreed on definitions of Agreed on definitions of PH practice Purpose Purpose Process Process Strengths Strengths Weaknesses Weaknesses

30. NPHPSP Instrument V2.0 should: Build on current instrument and Build on current instrument and lessons learned. Specify how users will validate Specify how users will validate their results. Recognize and respond to Recognize and respond to agency--system issues. Agree on purpose. Agree on purpose. Use for purpose for which designed. Use for purpose for which designed.