1. Federation of State Physician Health Program 2012 Annual Meeting FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 1

2. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 2 Betsy White Williams PhD MPH Assistant Professor Director of Outcomes and Research Office of Continuing Medical Education Rush University Medical Center Clinical Program Director Professional Renewal Center Lawrence, KS William H. Swiggart, MS, LPC/MHSP Assistant in Medicine Vanderbilt Department of Medicine Co-Director Center for Professional Health Vanderbilt University School of Medicine Nashville, TN Marine V. Ghulyan, MA Research Analyst The Center for Professional Health Vanderbilt University School of Medicine Nashville, TN Kayci Vickers Research Assistant Professional Renewal Center Lawrence, KS Michael V. Williams, Ph.D. Wales Behavioral Assessment Principal Wales Behavioral Assessment Lawrence, KS

3. Learning Objectives  Understand the results of 360 degree assessment in monitoring of intervention effectiveness.  Understanding the likely changes in the results from 360 degree evaluations over time.  Understanding the interpretation of outliers in utilizing a 360 degree evaluation to determine intervention efficacy FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 3

4. Presentation - Context  Physicians identified as disruptive practitioners are increasing being referred to Physician Health Programs. Heretofore it has been difficult to monitor the progress of these physicians post intervention.  This presentation discusses the development of a 360 survey instrument.  While we were interested in evaluating and comparing the data between cases and a comparison sample,  we were particularly interested in the use of the 360 for both identification and monitoring.  The 360° survey was recently developed based on input from experts and a review of the literature. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 4

5. 3 Core Competency Survey (3CC)  It is not enough to have good motives; others respond to our behavior.  Physicians are often not given essential feedback about their behavior.  The Three Core Competency Survey (3CC) is designed to provide feedback from those we work with. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 5

6. Disruptive behavior  “Intimidating and disruptive behaviors can foster medical errors, contribute to poor patient satisfaction and to preventable adverse outcomes, increase the cost of care, and cause qualified clinicians, administrators and managers to seek new positions in more professional environments.”  Issue 40: Behaviors that undermine a culture of safety | Joint Commissionhttp://www.jointcommission.org/assets/1/18/SEA_4 0.PDF FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 6

7. Aggressive Anger Outbursts Profane/Disrespectful Language Throwing Objects Demeaning Behavior Physical Aggression Sexual Comments or Harassment Racial/Ethnic Jokes PassiveAggressive Derogatory comments about institution, hospital, group, etc. Refusing to do tasks Passive Chronically late Not responding to call Inappropriate or inadequate chart notes Spectrum of Disruptive Behaviors FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 7

8. Disruptive behavior  “The term “disruptive behavior” is changed in the standards  The term “disruptive behavior” in two elements of performance (LD.03.01.01, EPs 4 and 5) has been revised to “behavior or behaviors that undermine a culture of safety.” ”  Joint Commission online November 11, 2011  http://www.jointcommission.org/assets/1/18/jconline_Nov_9 _11.pdf FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 8

9. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 9 Disruptive Behavior Leads to Communication Problems…Communication Problems Lead To Adverse Events 1  Communication breakdown factored in OR errors 50% of the time 2  Communication mishaps were associated with 30% of adverse events in OBGYN 3  Communication failures contributed to 91% of adverse events involving residents 4 Gerald B. Hickson, MD James W. Pichert, PhD Center for Patient & Professional Advocacy Vanderbilt University School of Medicine 1. Dayton et al, J Qual & Patient Saf 2007; 33:34-44. 3. White et al, Obstet Gynecol 2005; 105(5 Pt1):1031-1038. 2. Gewande et al, Surgery 2003; 133: 614-621. 4. Lingard et al, Qual Saf Health Care 2004; 13: 330-334

10. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 10 Failure to Address Disruptive Conduct Leads To  Team members may adopt disruptive person’s negative mood/anger (Dimberg & Ohman, 1996)  Lessened trust among team members can lead to lessened task performance (always monitoring disruptive person)... effects quality and patient safety (Lewicki & Bunker, 1995; Wageman, 2000) Felps, W et al. 2006. How, when, and why bad apples spoil the barrel: negative group members and dysfunctional groups. Research and Organizational Behavior, Volume 27, 175-222.

11. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 11 Disruptive Behavior Creates  fear  confusion or uncertainty  vengeance vs. those who oppose/oppress them  hurt ego/pride  grief (denial, anger, bargaining)  apathy  burnout  unhealthy peer pressures  ignorance (expectations, behavior standards, rules, protocols, chain of command, standards of care)  distrust of leaders  dropout: early retirement or relocation  errors  disruptive behavior begets disruptive behavior Vanderbilt University and Medical Center Policy #HR-027

12. “The Perfect Storm” PhysicianHospital/Clinic The external system The internal system FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 12 Two systems interact Good skills Poor skills Functional & nurturing Dysfunctional

13. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 13 Etiologies  Institutional Factors  Scapegoats  System Reinforces Behavior  Individual Pathology may over-shadow institutional pathology

14. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 14 Etiologies  Personal Factors  Individual pathology  Life Stressors  Lack of knowledge and skills

15. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 15 2004 AAMC Council of Deans “ Physicians are often poorly socialized and enter medical school with inadequate social skills for practice.” “There is a growing body of literature documenting residency programs do not prepare resident physicians adequately for the practice of medicine.”

16. 16 FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 16 Potential Resources for Healthy Coping  Courses  Coaches, counselors  Comprehensive Evaluation  360° Evaluations  Risk Managers  Physician Wellness  Treatment Centers  Office of General Counsel  State BME  Professional Societies  QI Officers  EAP  Others  State Physician Health Program

17. CME Remediation of “disruptive behavior”  Development of insight  Development of Skills  Development of implementation strategy  Feedback and monitoring FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 17

18. 18 FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 18 General Trends  At 3 months, significant improvements in 20 of the 22 physicians  Increased motivating behaviors and motivating impact  Decreased disruptive behaviors and disruptive impact  Changes in behavior reported by “others” Samenow CP, Swiggart W, Blackford J, Fishel T, Dodd D, Neufeld R, Spickard A. A CME Course Aimed at Addressing Disruptive Behavior. Physician Executive; 34 (1) Jan/Feb 2008: 32-40.

19. 360 Survey to Provide Feedback and Monitor Behavior  BASED on CORE COMPETENCY AREAS  Interpersonal and Communication Skills  Professionalism  System based practice FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 19

20. Survey Development  Review of 300+ publications of the evidence focused articles on physician professional behavior ;  Abstraction of assessment items with a expert based Delphi process to yield candidates for the final scale. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 20

21. Survey Development  Interpersonal and Communication Skills  Uses verbal communication to provide appropriate feedback to others.  Makes others feel comfortable approaching to ask questions or make suggestions.  Communicates effectively with patients. © Williams, Swiggart, and Williams FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 21

22. Survey Development  Professionalism  Willingly performs all tasks, functions, or responsibilities that are typically expected of him/her.  Responds promptly to telephone and pages.  Reports timely to hospital/clinical duties.  Responds quickly and appropriately to administrative communications. © Williams, Swiggart, and Williams FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 22

23. Survey Development  System-based practice  Creates a sense of teamwork and valued contribution by team members.  His/her behavior makes others comfortable in their work.  Creates an accepting work environment. © Williams, Swiggart, and Williams FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 23

24. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 24 Importance of Monitoring  Necessity of ensuring the behavior does not recur,  Anecdotal evidence of a significant level of recidivism,  Prior behavioral issues are a significant risk factor for later disruption ( Papadakis and colleagues, see for example, Papadakis, Arnold et al. 2008)  Facilitates earlier identification

25. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 25 Importance of Feedback  Form of guided self-assessment,  Reinforces behavioral changes,  Provides a standard by which to assess gains.

26. Methods-Subject Group  Participants in the The Course for Distressed Physicians, a remedial CME course developed at the Center for Professional Health at Vanderbilt University.  Cases: referred for workplace difficulties that relate to team behavior  Comparison: Physicians of similar specialties as the cases. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 26

27. Methods-Data Analysis  Specificity and sensitivity using the two classes of participants were analyzed.  The means and distributions were analyzed for consistency with other measures of performance more consistent with process measurement.  The outcome of an analysis of outliers is reported. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 27

28. Reliability and Validity  Neither reliability or validity is a simple issue in the context of measures across time  Consider the issue of the WAIS, while valid and reliable, serial tests can be problematic due to lagged time effects;  In the case of “disruptive behavior”, serial measures are core to the value of the assessment instrument. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 28

29. Measures of Validity  Types of validity:  Face validity;  Construct validity;  Predictive validity  In general these are summarized by Messick:  “ … the degree to which the empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores.”  Expanded by Kane to four domains:  Scoring, generalization, extrapolation, and interpretation/decisions. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 29

30. Today’s focus  While we touch briefly on our first two elements:  Scoring, and,  Generalization.  Most of the focus of this discussion is on:  Extrapolation, and,  Interpretation and decisions.  The interpretation and decisions element most clearly differentiate useful instruments from interesting academic exercises FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 30

31. Scoring  An extension of face validity  Is the item appropriate to the construct of interest;  We selected a scale shown to be valid in other 360 medical applications FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 31

32. Generalization  Convergence:  Our scale(s) generate a Cronbach’s α in excess of 0.9 the degree of exceeding depending on the item set.  Our scales, using underlying factor structures, demonstrate factor invariance across at least 3 sets of raters  The degree of coherence is clear, individual differences – the contribution to formal error – are being examined but two seem theoretically appropriate: time of remedial training and identification as disruptive. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 32

33. Extrapolation  Extrapolation – Do the scores predict real- world outcomes of interest, a broad restatement of predicative validity.  Four groups  distressed physician class participants at Vanderbilt  Distressed physician class participants at PRC  Comparison sample at Vanderbilt  Comparison sample from PRC.. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 33

34. Extrapolation No significant different was found between the two comparison samples. The data were pooled in subsequent analyses FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 34

35. Extrapolation The 360 demonstrates known group discrimination: The method is valid as far as discriminating between professionals identified as demonstrating behavioral issues. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 35

36. Extrapolation Interprofessional Behavior Note the reduction in mean difference across time is minimal; However, the reduction in variance, and particularly skew is marked. Extreme performance, particularly low performance is reduced over time. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 36

37. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 37 Disruptive behavior – Social systems  Preliminary results suggest that disruptive physicians may not differ significantly from normal physicians in mean performance but may differ significantly in skew. 37

38. Extrapolation Wave analysis for inter-professional behavior Again the change is mostly in the reduction of variance across time, not in means. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 38

39. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 39 Disruptive Behavior and Institutional Functioning  The presence of the system disruption ultimately results in breakdown:  Communications;  Affiliation;  Roles; and,  Protocols and duties. Comparison Department Index Department Index Physician Separation 39

40. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 40 Disruptive behavior – Social systems  Results suggest that disruptive physicians may not differ significantly from normal physicians in mean performance but may differ significantly in skew.

41. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 41 Disruptive behavior – Social systems  Over time as the disruptive behavior is extinguished the pattern of data will modify.  Results suggest the proportion of extreme reports falls and moderate to good reports increase.

42. Competencies and Team Function As outlying performance decreases team cohesion increases. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 42

43. Discussion  The instrument discriminates between participants in the remediation exercise and the comparison sample.  The instrument shows appropriate sensitivity and specificity and appears to be valid.  Analysis of outliers and serial results  means and distributions appear to be consistent with expectation  means and distributions change over time appropriately. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 43

44. Discussion  The tail of the distribution  appears very sensitive to behavioral change as reported by other observers.  The 3C 360° survey is a promising measure of CME efficacy in changing practice patterns. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 44

45. Interpretation/Decisions  The data support the discrimination between:  Physicians identified as being disruptive, and,  Physicians from comparison samples.  As well as:  Physicians identified as being disruptive, and,  Physicians in a general remediation program.  These data suggest that general interpretation is valid. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 45

46. Interpretation/Decisions  The instrument is intended to address:  Application within referral sources (hospitals, clinics, academic medical centers); and,  A methodology that both tracks improvement and indicates adequate performance.  These elements are met through a core competency structure and sensitivity to changing outliers. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 46

47. Discussion  The instrument  provides a consistent measurement with the literature and our experience of those areas of functioning related to interpersonal skill and communications, professionalism, and team behavior for healthcare professionals.  demonstrates appropriate sensitivity and specificity  provides the basis of effectively assessing intervention efficacy.  shows promise as a monitoring instrument and as a mean of identifying relapse behaviors. FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 47

48. Discussion  Sample Case Report FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 48

49. Discussion  Sample Comment page FSPHP April 23-26, 2012Williams BW, Swiggart WH, Ghulyan MA, Vickers, K, Williams MV 49