1. The Development of Quality Indicators for High Acuity Pediatric Conditions: Challenges in the Translation of Knowledge into Performance Measurement Antonia Stang MDCM MBA MSc Assistant Professor University of Calgary Departments of Pediatrics and Community Health Sciences

2. Disclosure I do not have an affiliation (financial or otherwise) with any commercial organization that may have a direct or indirect connection to the content of my presentation.

3. “If we want healthy citizens-as opposed to citizens who have ready access to sickness care-we need a profound philosophical shift in what we should expect from medical professionals. We need to reward and incent quality, not quantity” Andre Picard, Globe and Mail, March 20, 2012

4. Background Quality of Care: “the degree to which health services for individuals increases the likelihood of desired health outcomes and are consistent with current professional knowledge” (Institute of Medicine)

5. Background Quality Indicators: explicitly defined and measurable items pertaining to the structures, processes or outcomes of care –Structures: staff, equipment, physical layout of the department, laboratory and diagnostic imaging resources –Processes: interactions between professionals and patients –Outcomes: mortality, morbidity, patient satisfaction, quality of life

6. Quality Indicator Uses Improve health care and outcomes Benchmark performance Set minimum standards of care Improve efficiency Accountability Transparency Research Pay-for-Performance

7. What Makes a Good Measure? Impact, Opportunity, Evidence— Important to Measure and Report Reliability and Validity—Scientific Acceptability of Measure Properties Usability Feasibility National Quality Forum Measure Evaluation Criteria http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx

8. Objectives To review methods for involving stakeholders in the indicator development process. To discuss the application of GRADE (the Grading of Recommendations Assessment, Development and Evaluation) in indicator development and selection. To describe the challenges in developing and testing broadly applicable performance measures for high impact, relatively low frequency, conditions.

9. Study Objective to use a systematic process involving multiple stakeholders to develop evidence based quality of care indicators for pediatric conditions requiring high acuity ED care.

10. Research Team Antonia Stang MD MBA MSC, Principal Investigator Astrid Guttmann MD MSc, Co-Investigator David Johnson MD, Co-Investigator Sharon Straus MD MSc, Co-Investigator Lisa Hartling MSc PhD, Collaborator/methodological expert Francois Belanger MD, Collaborator/decision maker Angelo Mikrogianakis MD, Collaborator/decision maker Jen Crotts RN, Research Assistant Janie Williamson RN, Pediatric Emergency Research Team Coordinator

11. Funding Funded by an operating grant from the Canadian Institutes of Health Research (CIHR)

12. Rationale Lack of research on quality indicators specific to the pediatric population. Quality measures that are part of pediatric emergency medicine practice have not been systematically validated. Performance measures specific to pediatrics and pediatric emergency medicine have been identified as a research priority.

14. Stakeholder Involvement Goals: –to represent different stakeholder perspectives –to incorporate scientific evidence and expert opinion

15. Stakeholder Involvement Systematic methods to combine expert opinion and medical evidence –Consensus development conferences –Guideline based –Delphi technique –Nominal group technique –RAND/UCLA appropriateness method

16. Phase 1 Condition Selection 32 Member advisory panel Data on the main diagnosis for high acuity (resuscitation and emergent at triage) pediatric patients (age 0-19 yrs) seen in all EDs in Ontario and Alberta. Criteria for Condition Selection; –importance (morbidity or mortality) –impact (potential to address gap between current and best practice) –validity (adequacy of scientific evidence linking performance of care to patient outcome)

17. Phase 1: Condition Selection Table 1: High Acuity (Resuscitation and Emergent ) Pediatric visits in 2006/2007 and 2007/2008 for all EDs in Ontario and Alberta for Selected Conditions

18. Phase 2: Indicator Development Systematic Review of the Literature for each condition –Existing Indicators –High quality national and international guidelines (AGREE), Systematic Reviews (AMSTAR), Randomized Controlled Trials Criteria for Indicator Development –High quality evidence linking care structure or process to patient outcome –Strongly recommended –Consistency across guidelines

19. Literature Review

20. GRADE Grading of Recommendations Assessment, Development and Evaluation 1-Very low quality: Any estimate of effect is very uncertain 2-Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate 3-Moderate Quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate 4-High Quality: Further research is very unlikely to change our confidence in the estimate of effect

22. GRADE Inter-rater Reliability * Cohen’s Kappa unweighted

23. Challenges using GRADE Subjective Lack of concordance between guidelines Time consuming Requires substantial knowledge of clinical condition and research methodology Variable inter-rater reliability

24. Phase 3: Indicator Selection Expert Panel Process: 2 rounds of a web-based survey and a face-to-face meeting Indicators were selected based on two criteria rated on a scale of 1(strongly agree) to 9 (strongly disagree): –Relevance –Impact Indicators rated ≥7 on both criteria by 70% of panelists were retained

25. Relevance Impact IndicatorTypeSourceGRADENumeratorDenominator % of patients with anaphylaxis with documentation of specialist referral P25, 26, 28-31 2Number or patients with anaphylaxis with documentation of specialist referral including primary MD follow-up for referral, or documentation of existing specialist relationship Total number of patients with anaphylaxis (based on ICD- 10 codes)

27. Challenges in Indicator Development Lack of high quality evidence on the link between treatment/processes and outcomes, particularly in the pediatric setting Difficulty in identifying performance measures applicable to all settings

28. "The only man I know who behaves sensibly is my tailor; he takes my measurements anew each time he sees me. The rest go on with their old measurements and expect me to fit them." George Bernard Shaw

29. Phase 4:Data Collection *Based on ICD-10 code

30. Diabetic Ketoacidosis *Includes only visits with fluids or insulin started in study ED

31. Anaphylaxis * for food and insect sting induced reactions

32. Feasibility and Reliability *Number of ED visits based on ICD-10 code

33. Challenges in Indicator Testing Feasibility and Reliability –Retrospective –Proper diagnosis is in itself a quality issue –Accuracy of ICD -10 codes –Cost/effort of data collection Small and variable sample size

34. Lessons Learned Allow ample time for systematic review and evidence grading Composition of expert panel is key Need an experienced moderator Formal qualitative analysis of expert panel meeting Include patient/care-giver perspective

35. Next Step Multicentre data collection on select high acuity indicators combined with existing pediatric and emergency department performance measures –Reliability and feasibility testing –Process to outcome link

36. Campbell 2003

37. Quality Improvement and Indicator Development Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academy Press. 2001: Washington, DC. Campbell, S.M., et al., Research methods used in developing and applying quality indicators in primary care. BMJ, 2003. 326(7393): p. 816-9. Chassin MR et al. Accountability Measures-Using Measurement to Promote Quality Improvement. N Eng J Med 363:7. Donabedian A. The quality of care. How can it be assessed? JAMA. Sep 23-30 1988;260(12):1743-1748. http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx Center for Health Policy/Center for Primary Care and Outcomes Research & Battelle Memorial Institute. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement (Prepared by Battelle, under Contract No. 290-04-0020). Rockville, MD: Agency for Healthcare Research and Quality. May 2011.

38. Pediatric and Emergency Indicators Alessandrini E, Gorelick MH, Shaw K, Kennebeck S. Using Performance Measures to Drive Improvement in Pediatric Emergency Care 2010; http://webcast.hrsa.gov/postevents/archivedWebcastDetail.asp?aeid =534 http://webcast.hrsa.gov/postevents/archivedWebcastDetail.asp?aeid =534 Bardach NS, Chien AT, Dudley A. Small Numbers Limit the Use of the Inpatient Pediatric Quality Indicators for Hospital Comparison. Academic Pediatrics, 2010. 10(4). Guttmann A, Razzaq A, Lindsay P, Zagorski B, Anderson GM. Development of measures of the quality of emergency department care for children using a structured panel process. Pediatrics, 2006. 118(1): p. 114-23. Schull MJ, Guttmann A, Leaver CA, Vermeulen M, Hatcher CM, Rowe BH, Zwarenstein M, Anderson GM. Prioritizing performance measurement for emergency department care: consensus on evidence based quality of care indicators. CJEM 2011. 13(3):300- 309.

39. Rating the Quality of Evidence http://www.gradeworkinggroup.org/index.htm (accessed February 23, 2012)http://www.gradeworkinggroup.org/index.htm Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schünemann HJ; GRADE Working Group. Rating quality of evidence and strength of recommendations: What is "quality of evidence" and why is it important to clinicians? BMJ. 2008 May 3;336(7651):995-8.BMJ. 2008 May 3;336(7651):995-8 Guyatt GH, Oxman AD, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ, for the GRADE Working Group. Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-926.BMJ 2008;336:924-926 Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW Jr, Kunz R, Craig J, Montori VM, Bossuyt P, Guyatt GH; GRADE Working Group. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008 May 17;336(7653):1106-10.BMJ. 2008 May 17;336(7653):1106-10 Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009AGREE Next Steps Consortium (2009). The AGREE II Instrument [Electronic version]. Retrieved March 23 2012, from http://www.agreetrust.org.