1. Data Collection and Measurement Diane Abatemarco, PhD, MSW Ruth S. Gubernick, MPH Practicing Safety Learning Session May 30, 2009

2. Disclosures Diane Abatemarco :I have no relevant financial relationships with the manufacturers of any commercial products and/or provider of commercial services discussed in this CME activity. I do not intend to discuss an unapproved/investigative use of a commercial product/device in my presentation. Ruth Gubernick:I have no relevant financial relationships with the manufacturers of any commercial products and/or provider of commercial services discussed in this CME activity. I do not intend to discuss an unapproved/investigative use of a commercial product/device in my presentation.

3. Measurement, Data Collection & Evidence of Change  Importance of measurement – Why?  How will we know that a change is an improvement?  Clarify and be directly linked to goals  Seek usefulness over perfection  Be integrated into daily work whenever possible  Be graphically and visibly displayed  For PDSA cycles, be simple and feasible enough to accomplish in close time proximity to tests of change

4. How will we know that a change is an improvement? Plan DoStudy Act What are we trying to accomplish? What changes can we make that will result in improvement? The Improvement Model The Improvement Guide Associates in Process Improvement

5. Measurement for Quality Improvement  You can’t improve what you can’t (or don’t) measure  A good aim statement provides clear direction.  Measures tell a team if the changes they make are making a difference  Measurement tells you where you are and where you are going

6. Measures  Need to define Target population Numerator Denominator

7. Practicing Safety: Example  Target Population: All 2 month old infants seen in a participating practice for well care visit  Numerator: # of infants with documentation in chart that parent/caregiver received assessment/screening regarding coping with crying at or by the 2 month well visit  Denominator: All 2 month old infants seen in participating practice for well care visit whose charts are reviewed

8. Effective Measurement  Seek usefulness, not perfection  Keep measurement simple, think big, but start small

9. Effective Measurement: Outcomes Outcome measures: represents the voice of the customer or patient  Average hemoglobin A1c  Hospitalizations or ED visits due to asthma  Patient satisfaction with time to getting an appointment

10. Effective Measurement: Processes  Process measures: represents the workings of the system Percent of charts with documentation indicating use of 1 or more tools for maternal depression at or by the 2 month well visit. Percent of parents/caregivers receiving assessment/screening regarding discipline at or by the 18 month well visit.

11. Effective Measurement  Build measurement into daily work routine Data should be easy to obtain and timely Small samples over time  Use quantitative and qualitative data Qualitative data is highly informative Qualitative data is easy to obtain

12. Why Plot Data Over Time  You develop a process to screen mothers for maternal depression using the Edinburgh Postnatal Depression Scale. The 6 months before implementing the process the average % screened using this tool is 10%. Six months after the process is implemented, the average % of mothers screened is 90%.  How will you answer the question: was this change an improvement?

13. Change Run Charts

14. Change

17. Chart Documentation Form and Instructions  Include in charts; not used as retrospective chart review form  Use in charts of specified visits (2 month; 18 month; or other as determined by your team)  Leave form in chart to document follow-up  Insert practice ID numbers on Charts Documentation Forms before making copies for charts  Submit on 30 th of month to Ruth  Note: Binders include 1 form for each age group per month (totaling 12 forms)

18. Chart Documentation Form and Instructions  Team Planning Identify which of your infant and toddler patients will be seen for well care during each month Use Chart Documentation Form included as part of their pre-visit planning/preparation This might change over the six (6) month period, as your team conducts PDSA cycles to test the process and the Toolkit  Your team will also decide how you will determine which 10 Infant Chart Documentation Forms and 10 Toddler Chart Documentation Forms will be submitted to the Practicing Safety Project team for review. For example:  Your protocol may be to submit copies of the Chart Documentation Forms for those infant and toddler patients seen for a well-child visit one day each week (i.e., each Monday or Friday) that month  Or those infant and toddler patients seen for well-care during the last week of that month, for a total of 10 forms of each cohort.

19. Example Run Charts from Chart Documentation Form

22. Monthly Progress  Provides information about Tests of change completed each month Assessment of team progress Other qualitative measures  Each month, staff will email the Monthly Progress Report so it can be completed electronically Instructions  Insert your practice’s Aim statement.  Indicate the Practicing Safety bundles you have implemented. Describe specific changes and tools you have tested.  Rate your team’s progress using the scale. Report your team’s learning.  Complete Index.  E-mail your monthly progress report to Jill Healy at jhealy@aap.org jhealy@aap.org

23. Example Chart from Monthly Progress Report Index

24. Project Workspace  All data and run charts will be available for review on the Project Workspace Web site http://www.aap.org/qualityimprovement/qui in/workspaces/PracticingSafety/PS_Home. html http://www.aap.org/qualityimprovement/qui in/workspaces/PracticingSafety/PS_Home. html Username: pracsafety Password: r3f5y7  Compare your team’s data to other teams and to the aggregate