1. "Improving Improvement: Building and Sharing Best Evidence for Clinical Decision-making" Paul Batalden, M.D. Stephen Liu, M.D. Dominic Geffken, M.D. Travis Harker, M.D. (Frank Davidoff, M.D.) Dartmouth-Hitchcock Leadership Preventive Medicine Residency Program Center for the Evaluative Clinical Sciences Dartmouth Medical School February, 2006

2. Acknowledgements Frank Davidoff David Stevens Don Berwick Gene Nelson Gerry O’Connor Mark Splaine Stephen Plume Margie Godfrey Wayne Dysinger Carolyn Murray Marvin Dunn Tina Foster Greg Ogrinc David Aaron Dan Eubank Tom Colacchio Paul Gardent George Anderson Jeff Davis Linda Kinsinger Bob Harmon David Leach & Many others…

3. Session Objectives: 1.Describe the shortcomings of current publications of quality improvement work. 2.List the key elements and criteria necessary for the publication of quality improvement work. 3.Identify how use of the elements and criteria described in this session will help improve your own improvement work and associated publications. 4.Illustrate how these understandings can integrate with a Preventive Medicine residency program.

4. Aim To preview and illustrate the knowledge systems involved in improving patient care…and their implication for publication and for residency education.

5. Assumptions We know that clinical care could be better than it is—and it’s frustrating to keep getting that news. We all want to improve clinical care. It is hard enough to improve personally, much less try to teach others how to change the world or the setting we work in—better to take up digital photography, woodworking or knitting. We are able and willing to suspend what we usually defend for a few minutes to explore some slightly different ways of thinking, if it’s interesting.

6. The way we use scientific evidence in the care of individual patients.

7. When we want to use evidence to improve a patient’s situation… Generalizable Scientific Knowledge Particular Patient Measurable Improvement +

8. The way we can use that same thinking about improving the care for a population of patients.

9. When we want to use evidence to improve the usual care for a population of patients… Generalizable Scientific Evidence Particular Context Measurable Performance Improvement +

10. Building knowledge of the particular context, setting, population Process? System failures? Habits? Traditions? Population? Professionals? Setting? Successful changes? Hot buttons? Frustrations? Social supports?

11. Population – Who are our CAP patients? All patients admitted to DHMC with a primary diagnosis of pneumonia from July 1 to September 30, 2003 –59 patients identified –11 excluded 6 charts not available 2 pediatric cases 3 no initial diagnosis of CAP

12. Population – Who are our CAP patients? Patient Age: –Average: 70 years (range 26-100 years) Length of Stay: –Average length of stay: 8 days (range 1-45 days)

13. Population – Where are CAP patients coming from? Emergency Department Outpatient Clinics Transfers From Outside Hospitals 63% 15% 22% DHMC Inpatient Ward

14. Professionals – Who takes care of CAP patients? General internal medicine (GIM) 51% Hematology/Oncology17% Cardiology10% Pulmonary10% Family practice 7% Nephrology 3% Rheumatology 2% Total 100% ICU/CCU admissions:10.4%

15. Patterns – How do we take care of our CAP patients? Emergency Department GIM Clinic Transfers From Outside Hospitals 3.5 hours – 67% within 4 hours 9.5 hours – 0% 10.4 hours – 0% Average time from patient arrival to first antibiotic therapy Percentage of patients receiving antibiotics within 4 hours

16. An Actual Patient Admission 2:00 PMThe patient arrives for her scheduled appointment. The provider performs a physical exam and orders lab tests and a chest x-ray. 2:45 PMThe patient reports to the outpatient lab 3 - 4:00 PMThe patient goes to the outpatient radiology department where she receives a chest x-ray 4:00 PMThe provider reviews the labs and the CXR. A diagnosis of pneumonia is made 5:00 PMAdmitting, the admitting attending and the resident team are called and notified of the admission 6:30 PM The patient arrives to her inpatient room on One East 7:05 PMThe admitting team writes admission orders for the patient 10:15 PMAzithromycin 500 mg IV is administered 12:00 AMCeftriaxone 1 gram IV is administered

17. Building knowledge of the CAP context, setting, population Habits? –Time had not been an important consideration in treatment for CAP Traditions? –Defer treatment decisions to the admitting residents for “teaching” purposes –Inpatient treatments were not usually given in the outpatient clinic

18. Building knowledge of the CAP context, setting, population Frustrations? –Significant delays in admitting patients from the ED and clinic were occurring due to a lack of inpatient bed openings –Loss of information and lapses in treatment were occurring during patient “handoffs” and transfers of care –Multiple admissions to an on-call team created delays in the admission process

19. Building knowledge of the CAP context, setting, population System Failures? –Antibiotics ordered as QD were given the morning after admission Not ordered as first dose now –Antibiotic change from a q8 to q24 medication resulted in inappropriate administration –Multiple doses of q24 antibiotics – in ED and on the floor within several hours

20. Building generalizable scientific knowledge A clear question? About the evidence: –Control for context? –Design tests for generalizability? –Careful search for biases? –Statistical analysis to guide inferences? Peer review prior to publication? Stored in accessible format(s)?

21. What is ideal care for CAP patients? Guidelines exist on “Ideal” Treatment for CAP (IDSA, AHRQ, ATS, ACEP) –Diagnostic Tests (Blood Cultures, Sputum GS) –Appropriate Antibiotics –Timely Antibiotic Treatment –Prevention Vaccinations Smoking Cessation Advice

22. Building generalizable scientific knowledge about CAP Timing of antibiotics matters –Patients who received antibiotics within 4 hours had an adjusted OR for mortality of 0.83 (0.72-0.96) –The mean length of stay was also 0.4 days shorter Houck PM, Bratzler DW. Timing of antibiotic administration and outcomes for Medicare patients hospitalized with pneumonia. Archives of Internal Medicine 2004; 164:637-644.

23. Building generalizable scientific knowledge about CAP Choice of antibiotics matters –Patients who receive an antibiotic regimen consistent with IDSA guidelines had a 26- 36% reduction in 30 day mortality rates Gleason, PP, Meehan TP. Associations between initial antibiotic therapy and medical outcomes for hospitalized elderly patients with pneumonia. Arch Int Med. 1999; 159:2562- 2572.

24. Building generalizable scientific knowledge about CAP Vaccinations are effective –Influenza vaccination has an efficacy of 53% for preventing pneumonia, 50% for preventing hospitalization, and 68% for preventing death Gross PA, Hermogenes AW. The efficacy of the influenza vaccine in elderly patients: a meta-analysis and review of the literature. Ann Int Med 1995; 123:518-27 –Pneumococcal vaccination is estimated to have a 47% efficacy for the prevention of pneumococcal bacteremia Shapiro ED, Berg AT, Austrian R, et al. The protective efficacy of polyvalent pneumococcal polysaccharide vaccine. N Engl J Med 1991; 325: 1453–60.

25. Measuring performance improvements Variation over time? Graphical displays? Balanced measures of outcome? Concurrent, real-time measurement? Measurement error control—operational definitions? Measurement design fidelity?

26. Process and Outcome Measures of Quality for CAP Percentage of patients receiving antibiotics within four hours of arrival to DHMC Percentage of patients receiving appropriate antibiotics Percentage of patients receiving blood cultures prior to antibiotics

27. Process and Outcome Measures of Quality for CAP Percentage of patients receiving oxygenation assessment on admission Percentage of eligible patients receiving a pneumococcal and influenza vaccination Percentage of eligible patients receiving smoking cessation advice or counseling Length of stay Mortality rate/Discharge disposition

28. Community-Acquired Pneumonia Performance Measures 2003 (%) Time to treat 62 Oxygenation 100 Blood cultures 62 Pneumovax 38 Smoking 49 Composite 10

29. Community Acquired Pneumonia Outcome Measures July – Sept 2003 N = 48 LOS - Mean 8 LOS - Median 5 Range 1 - 45

35. Getting the plan right for connecting generalizable scientific knowledge to particular context Standardization? Prompting strategies? Algorithms? Academic detailing? Resequencing work steps? Relationship work? Conflict management / negotiation strategies? Automation? Eliminate steps? Combine/couple steps? Change process ownership? Other? 1.Clarify aim(s) 2.Identify alternatives: 3. Select the best fit

36. Getting the plan right for connecting generalizable scientific knowledge to particular context Aim 1.Get correct treatment reliably started in less than four hours after arrival. 2.Prevent unnecessary CAP by improving immunization and smoking cessation care.

37. Alternatives for improving the care for CAP patients Start treatment where the patient is when diagnosis is made Develop standard admission orders and antibiotic guidelines Create documents that facilitate communication between outpatient and inpatient providers Include prompts for necessary vaccinations and smoking cessation in the admission orders Provide individualized feedback to providers about the care that they provide to pneumonia patients

38. “Making it happen” Local and non-local strategies of importance? “Ways” things get done? What matters to people who work here, how, why? Leadership of change history? Ways leaders manifest their support? Sustaining / embedding strategies? “Unlearning” needed? Supports for same? Relation of measurement to operations? Change ownership? Clinical policy ownership? Log of change process?

39. Knowledge of “how to make it happen” at DHMC “Ways” things get done? –Needed input and support from ED, GIM Clinic, One East, ID, Pharmacy, Nursing, and Residents –Needed approval of various committees for standard order forms –Needed support of administration –National and local priority

40. Knowledge of “how to make it happen” at DHMC What matters to people who work here, how, why? –Infectious Disease very concerned about possible misuse of standard orders have been frustrated in the past by the lack of diagnostic testing are primarily concerned with relatively rare causes of pneumonia

41. Knowledge of “how to make it happen” at DHMC What matters to people who work here, how, why? –Emergency Department Looking for ways to determine the appropriate antibiotics for CAP patients Have not traditionally used written orders

42. Knowledge of “how to make it happen” at DHMC What matters to people who work here, how, why? –General Internal Medicine Concerned about keeping up with busy schedules while treating patients who need to be admitted Increasing problem of patients waiting in the clinic for an inpatient bed

43. Was there any measurable improvement?

44. Community-Acquired Pneumonia Performance Measures 2003 (%) Time to treat 62 Oxygenation 100 Blood cultures 62 Pneumovax 38 Smoking 49 Composite 10

45. Community-Acquired Pneumonia Performance Measures 2003 (%) Jan – March 2005 (%) Time to treat 6281 Oxygenation 100 Blood cultures 6296 Pneumovax 3855 Smoking 4975 Composite 1038

46. Community Acquired Pneumonia Outcome Measures July – Sept 2003 N = 48 Jan – March 2005 N = 47 LOS - Mean 84.6 LOS - Median 54 Range 1 - 452 - 14

47. Knowledge systems involved Generalizable Scientific Evidence Particular Context Measurable Performance Improvement + control context out design for generalizability I focus on particular / identity II time measurement fidelity III right plan IV making it happen V

48. Brief break

49. Toward stronger evidence on quality improvement How can publication guidelines help? Davidoff, Batalden

50. Quality Improvement: A three component engine Scientific knowledge Competent individual and system performance Knowledge about how to acquire and increase competence Davidoff, Batalden

51. Learning is the source of energy that drives all quality improvement Davidoff, Batalden

52. QI: A three component engine ComponentLearning method Scientific knowledgeScientific discovery Competent individual and system performance Experiential learning Knowledge about how to acquire and increase competence Experiential discovery Davidoff, Batalden

53. Scientific discovery Goals –Primary: Produce and distribute new generalizable knowledge –Secondary: Improve performance Action cycle –Plan  Do  Study  Publish (Tom Nolan) Davidoff, Batalden

54. Experiential learning Goals –Primary: Acquire competence – personal or system ability to perform well –Secondary: Produce and distribute new generalizable knowledge Action cycle (basic) –Experience  Question (What?)  Conceptualize (Why?)  Retry Davidoff, Batalden

55. Experiential learning cycles General (informal) –Experience  Question  Conceptualize  Retry Special (formal) –Plan  Do  Study  Act Note: Publishing isn’t part of either cycle! Davidoff, Batalden

56. Experiential discovery Goals –Primary Produce and distribute new generalizable knowledge - specifically about experiential learning –Secondary Improve performance Action cycle –Plan  Do  Study  Publish (about experiential learning) Davidoff, Batalden

57. Why is publication important in all discovery (learning)? Logic: role in proof/disproof of efficacy Collective memory: establishes context Efficiency: prevents reinventing wheel, repeating mistakes Ethics: gives back information of value Accountability: openness allows public scrutiny Transparency: writing clarifies thinking Creativity: innovation sparks innovation Interaction: publishing and discovery reinforce each other Davidoff, Batalden

58. The discovery-publication interaction Researchers  publications – Content Publications  researchers – Critical review – Peer approval – Readability – Distribution system – Access (indexing, storage, retrieval) – Publication standards Davidoff, Batalden

59. Adding publication to QI General (informal) Experience  Question  Conceptualize  Retry | Teach/Coach Specific (formal) Plan  Do  Study  Act | Publish Davidoff, Batalden

60. Why isn’t QI work published more often? Done by busy “front line” professionals Lack of experience in doing studies, publishing Lack of academic incentives Writing is hard Editors, peer reviewers unfamiliar, skeptical Little guidance available on reporting Davidoff, Batalden

61. QI publication guidelines - background A work in progress Model from previous guidelines process: CONSORT, QUOROM, STARD Builds on guidelines for QI reports: QSHC, BMJ Davidoff, Batalden

62. QI Publication guidelines - features Purposes –Improve reporting: transparency –Improve the work, increase its use Users –Authors –Editors, reviewers, funders Applications –Original work –Formal > narrative –Reviews, commentary Features –IMRaD –Manageable number of items –Distribution of items Davidoff, Batalden

63. QI Publication Guidelines: next steps Improve: –Feedback, critique, especially from users –Stakeholder meeting –Evidence-based revision Expand: –Extensions, variants Research: –Item value –Impact on writing, review, editing Promote: –Plans for endorsement, adoption, diffusion Educate: –Teaching sessions –Explanation and elaboration document (a la CONSORT) Davidoff, Batalden

64. Summary QI: a complex mix of knowledge, performance Driven by hybrid of learning modes: –scientific discovery –experiential learning –experiential discovery Publication: essential in scientific discovery, little impact yet in QI Publication guidelines: could support more and better QI reporting Davidoff, Batalden

65. Let’s try this out…

66. Guidelines testing exercise 1.Pick a partner to work with. 2.Review the guideline reference sheet. 3.Select an article. 4.Markup the article, noting the guideline number on the text. 5.Be prepared to comment on the work.

67. Now… Reflect on what you just did… –How were the guidelines helpful? –Things missing? –Did you and others disagree about how to apply guideline items to the article? –How were they unclear? –How might they be made more useful?

68. Together… So, let’s hear about it So now what? If you want your thoughts to join the development process, make sure your comments on the worksheets are legible and your contact information is clear on the worksheet and give them to us.

69. Brief break

70. Helping residents learn using this frame…

71. Dom’s slides

72. Reactions, comments, questions?