Presentation on theme: "Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior Philip Hemphill, Ph.D. Professional Enhancement."— Presentation transcript:
Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior Philip Hemphill, Ph.D. Professional Enhancement Program (PEP) Pine Grove Behavioral Health and Addictive Services firstname.lastname@example.org
FSPHP Presenter Disclosure Slide “Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior” Philip Hemphill, PhD is a paid consultant of Pine Grove Behavioral Health and Addiction Services FSPHP Annual Conference and Meeting Fort Worth, Texas April 23-26, 2012
Why use a MSF/360? 1. Facilitate insight of strengths and weaknesses for an individual. 2. Enhance organizational cultural change. 3. Provide summative assessment of performance. 4. Evaluate potential. 5. Improve team effectiveness. 6. Identify training & coaching needs. 7. Make organizational values explicit. 8. Measure progress after training & coaching.
MSF/360 Essentials Proven validity & reliability. Useful in assessing “humanistic” skills. Weak or no associations between self-rated assessment and external assessment. “…methods such as multisource feedback (360) evaluations may be a necessary next step, particularly when interpersonal, communication skills, or professionalism needs to be evaluated (p. 1101).” Source: Davis et al. (2006). Accuracy of physician self-assessment compared with observed Measures of competence.JAMA, 296 (9), 1094-1102.
Key Players in MSF/360 Target Individual Raters Boss FacilitatorVendor
MSF/360 Internal Process Feedback Is it consistent with self-perception? Emotional Response Reflection and internal processing of emotion and content. Decision to accept and act, or not Action for learning, change Source: Sargeant et al. (2009) Advances in Health Sc Ed.
Relevant Literature Review of Multi-Source Feedback Measures The Fundamental Interpersonal Relations Orientation–Behavior™ (FIRO-B®) In the late 1950s William Schutz, PhD. developed the FIRO-B theory to aid in the understanding and predicting of how high- performance military teams would work together. In developing the FIRO-B theory, Schutz began with the premise that "people need people." He used the term interpersonal to indicate any interaction, real or imagined, occurring between people. He used the term need to describe a psychological condition that, if not satisfied, leads to a state of discomfort or anxiety. He posited that interpersonal needs could be grouped into three categories: Inclusion, Control, and Affection. The FIRO-B model describes the interaction of these three categories of interpersonal need along two dimensions: expressed and wanted. Awareness Scales and were designed primarily to help individuals with their self-awareness and of understanding their relation to other people (Schutz, 1978).
Relevant Literature Review of Multi-Source Feedback Measures 1. Developing insight into strengths and weaknesses, for the individual (Farh & Dobbins, 1989; Hazucha et al., 1993; Martocchio & Judge, 1997; Fletcher, 1999; Keeping et al., 1999; London et al., 1999; McCarthy & Garavan, 1999) and the organization (London & Beatty, 1993; Lepsinger & Lucia, 1998; Steensma et al., 1998; Tornow & London,1998) to act upon. 2. Enhancing culture change (Cardy & Dobbins, 1994; O’Reilly, 1994; Crossley & Taylor, 1995; Church & Bracken, 1997; Bracken et al., 1998; DeSimone, 1998). 3. Summative assessment of performance (Newble et al., 1999). 4. Evaluating the potential of individuals, for instance to use in career advice or selection (Tornow, 1993; Lindsey et al.,1997; Facteau et al., 1998). 5. Enhancing team effectiveness (by allowing opportunity to comment) (Garavan et al., 1997; Towers-Perrin, 1998). 6. Identifying training needs for the system (as opposed to the individual) (Towers-Perrin, 1998).
Development of Disruptive Behavior Instrument 2004 – Developed list of disruptive behaviors and boundary violation behaviors. Approximately 70 items… 5 people monitored over 12 months – Instrument too long; complaints from participants; descriptive data…. 2005- Cut instrument down to 50 items(1 Fctr) 2008- Cut down to current version Been using current version for past 4 years…
Definition of Disruptive Behavior Disruptive behavior is any inappropriate behavior, confrontation, or conflict, ranging from verbal abuse to physical or sexual harassment. Disruptive behavior causes strong psychological and emotional feelings, which can adversely affect patient care. Rosenstein A, O’Daniel M. (2008). Managing disruptive physician behavior: Impact on staff relationships. Neurology, 70, 1564-1570.
Disruptive Behavior Items Avoids taking responsibility for actions Inappropriately blames others Places other interests above the safety and welfare of community members Suspects, without sufficient basis, that others are exploiting, harming, deceiving, or out to “get” him Seems to have a preoccupation with unjustified doubts about the loyalty or trustworthiness of others Reads hidden, demeaning, or threatening meanings into benign remarks or events Perceives attacks on his character/reputation not apparent to others and is quick to react angrily/counterattack Shows rigidity and stubbornness Displays an unpredictable mood Responds to constructive criticism in an immature/irresponsible fashion Takes too long to requests from others Arrives late Fails to complete paperwork in a timely manner Responds to requests in an avoidant, unreliable, and/or uncooperative manner Displays a sense of entitlement Shows perfectionism that interferes with task completion Behaves in a sexually inappropriate manner (verbally or physically) Lacks empathy Intimidates others or attempts to intimidate others Exhibits physically aggressive or assaultive behaviors Exhibits arrogant behaviors and/or has an arrogant attitude Demonstrates an inflated sense of self-importance Takes advantage of others in order to get what he wants Is reluctant to delegate tasks or to work with others unless they submit to exactly his way of doing things Expresses anger/hostility inappropriately Exhibits passive-aggressive behavior Uses abusive, belittling, condescending, demeaning, and/or threatening language
Client signs release allowing for electronic correspondence, and client given monitoring instructions and materials with due date. Client identifies MC, gives MC instructions and rating materials, and with MC identifies approx. 15 raters.. Ratee email addresses Collection of surveys online. Rating process is repeated at predetermined intervals until four uneventful reports Client, MC, and other relevant stakeholders review the feedback report Significant impairment is present and PHP/LAP intervenes CLIENT SIGNS/ UPDATES MONITORING CONTRACT WITH PHP/LAP AND/OR No significant impairment is present Client attends treatment and rating process is temporarily suspended PHP/LAP recommends interventions Client participates in interventions while continuing rating process PHP/LAP recommends intensive/ residential treatment Client completes treatment Report produced and emailed to persons indicated on release. MONITORING RATING FLOWCHART
33 questions 28 Likert-type questions regarding ratee’s behavior 1 question concerning how many hours rater works with ratee 1 question about the rater’s title 3 open-ended questions that solicit comments
Survey Dimensions Appropriate Documentation Arrogance Citizenship Compliance Empathy Exploitation of Others Expression of Anger Intimidation Job Attitudes Multicultural Sensitivity Openness to Others Perfectionism Respect for Others Response to Criticism Responsibility for Actions Rigidity Satisfaction with Ratee Sexual Appropriateness Stability of Mood Stress Management Supervision Suspiciousness Tardiness Team Impact Threatening Behavior Verbal Aggression Working with Others
Disruptive Behavior Instrument Descriptive Characteristics of Sample: N = 22 ratees (498 surveys) Average Age = 38 years Male = 19 ; Female = 3 Survey Breakdown by rater: Self-Report = 53 Staff = 328 Supervisor = 37 Peer = 79 Number of Times each patient was assessed: Range – 1-8 times
Disruptive Behavior Instrument Aggression Aggression: Non-verbal Aggression: Physical Aggression: Verbal Aggression: Written Aggression Aggression: Non-verbal Aggression: Physical Aggression: Verbal Aggression: Written Interpersonal Difficulties Intimidation Lack of Empathy Lack of Multicultural Sensitivity Lack of Openness to Others Problematic Perfectionism Response to Criticism Rigidity Suspiciousness Interpersonal Difficulties Intimidation Lack of Empathy Lack of Multicultural Sensitivity Lack of Openness to Others Problematic Perfectionism Response to Criticism Rigidity Suspiciousness Sexually inappropriate behavior Sexual Inappropriateness Sexual/Inappropriate Remarks Sexually inappropriate behavior Sexual Inappropriateness Sexual/Inappropriate Remarks Self Regulation Mood Instability Poor Stress Management Self Regulation Mood Instability Poor Stress Management Disruptive Behavior Arrogance Blaming Others Disobeying Rules Disrespect for Staff/Colleagues Ease of Working with Ratee Employee Turnover Intentions Exploitation of Others Poor Citizenship Tardiness Disruptive Behavior Arrogance Blaming Others Disobeying Rules Disrespect for Staff/Colleagues Ease of Working with Ratee Employee Turnover Intentions Exploitation of Others Poor Citizenship Tardiness Originally hypothesized to be based on 5 factors
Disruptive Behavior Instrument Total Variance Explained FactorExtraction Sums of Squared LoadingsRotation Sums of Squared Loadings Total% of VarianceCumulative %Total 110.46338.750 9.296 21.8366.80145.5517.498 31.4085.21550.7661.655 Extraction Method: Principal Component Analysis. a. When components are correlated, sums of squared loadings cannot be added to obtain a total variance.
Disruptive Behavior Instrument Component Correlation Matrix Factor123 11.000-.5431.35 2-.5431.000-.031 3.135-.0311.000 Extraction Method: Principal Component Analysis. Rotation Method: Promax with Kaiser Normalization.
Disruptive Behavior Instrument Cronbach’s Alpha ( α ) N of Items Factor 1.92213 Factor 2.8799 Factor 3.3054 Reliability for Three Factors Poor reliability for items on Factor 3 suggests that a 2 Factor structure may be a better fit.
Disruptive Behavior Instrument 2 Factors Interpersonal Aggression Aggression: Non-verbal Aggression: Physical Aggression: Verbal Aggression: Written Arrogance Blaming Others Disobeying Rules Exploitation of Others Intimidation Lack of Multicultural Sensitivity Poor Citizenship Suspiciousness Interpersonal Aggression Aggression: Non-verbal Aggression: Physical Aggression: Verbal Aggression: Written Arrogance Blaming Others Disobeying Rules Exploitation of Others Intimidation Lack of Multicultural Sensitivity Poor Citizenship Suspiciousness Professionalism Disrespect for Staff/Colleagues Lack of Empathy Lack of Openness to Others Mood Instability Poor Stress Management Problematic Perfectionism Response to Criticism Rigidity Tardiness Professionalism Disrespect for Staff/Colleagues Lack of Empathy Lack of Openness to Others Mood Instability Poor Stress Management Problematic Perfectionism Response to Criticism Rigidity Tardiness
Disruptive Behavior Instrument Instead of a third factor, items were named “critical items”. 5 Critical Items: “I think about quitting or transferring to a different job because of the ratee.” “Touches or looks at one or more patients/coworkers in an inappropriate/sexually suggestive manner.” “Makes sexual remarks in the workplace that are inappropriate or tells jokes that makes others uncomfortable.” “Performs unnecessary exams on female and/or male patients.” “Demonstrates lax boundaries when prescribing controlled substances, giving out samples, or storing samples of controlled substances.”
Disruptive Behavior Instrument Conclusions/Limitations: We have 2 main factors and 5 critical items. Two factors: Interpersonal Aggression Professionalism
Future Directions: Measuring and refining Inter-rater Reliability Measuring and refining Test-Retest Reliability How is this data utilized? What are cutoff scores? Can we track disruptive behavior without treatment? Can instrument be utilized outside of medicine?
We do not deal much in fact when we are contemplating ourselves. Mark Twain