Neal D. Kohatsu, MD, MPH Medical Director May 17, 2012 Evaluation and Quality Improvement

2 What is Evaluation? “The systematic investigation of the merit, worth, or significance of an ‘object’.” Michael Scriven “…the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards…” Carol Weiss “A systematic way to determine the “value” of a program, program components, or activity.” cited by Ann Powers

3 “Research seeks to prove, evaluation seeks to improve…” M.Q. Patton

CDC Evaluation Framework 4

5 Steps in Program Evaluation Step 1: Engage Stakeholders Step 2: Describe the Program Step 3: Focus the Evaluation Design Step 4: Gather Credible Evidence Step 5: Justify Conclusion Step 6: Ensure Use and Share Lessons Learned MMWR; Vol. 48; 9/17/99

6 The Four Standards Utility: Who needs the information and what information do they need? Feasibility: How much money, time, and effort can we put into this? Propriety: What steps need to be taken for the evaluation to be ethical? Accuracy: What design will lead to accurate information? MMWR; Vol 48; 9/17/99

Quality Improvement 7

Shewart/Deming Cycle 8

Rapid-Cycle QI Accelerates system change Smaller change cycles and samples Greater focus on improvement (vs. measurement) 9

Approach to Rapid Cycle QI Determine specific aim Define measurement Specify intervention 10

Conclusions Evaluation and QI are related. Must adapt to a system context. Foster a learning environment. Encourage collaboration. Focus on outcomes. 11